Content movement and interaction using a single controller

ABSTRACT

Examples of systems and methods for interacting with content and updating the location and orientation of that content using a single controller. The system may allow a user to use the same controller for moving content around the room and interacting with that content by tracking a range of the motion of the controller.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Pat. Application No.17/15320, filed on Jan. 20, 2021, entitled “CONTENT MOVEMENT ANDINTERACTION USING A SINGLE CONTROLLER,” which claims the benefit ofpriority under 35 U.S.C. § 119(e) to U.S. Provisional Application No.62/965708, filed on Jan. 24, 2020, entitled “CONTENT MOVEMENT ANDINTERACTION USING A SINGLE CONTROLLER,” the disclosures of each of whichare hereby incorporated by reference herein in their entirety.

FIELD

The present disclosure generally relates to systems and methods tofacilitate interactive virtual or augmented reality environments for oneor more users.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality”, “augmentedreality”, or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects coexist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR, and MR technology.

SUMMARY

Embodiments of the present disclosure are directed to devices, systems,and methods for facilitating virtual or augmented reality interactionfor one or more users.

Further details of features, objects, and advantages of the disclosureare described below in the detailed description, drawings, and claims.Both the foregoing general description and the following detaileddescription are exemplary and explanatory and are not intended to belimiting as to the scope of the disclosure.

In some configurations, an augmented reality (AR) system can include: anAR display configured to present virtual content to a user of the ARsystem; an outward facing camera configured to capture one or moreimages of an environment of the user; a handheld controller defining apointing vector indicating a pointing direction of the handheldcontroller; a hardware processor in communication with the AR display,the outward facing camera, and the handheld controller, the hardwareprocessor can be programmed to: display an interactive content objectvia the AR display; in a first interaction mode, while the pointingvector remains within the interactive content object, indicatingmovements of the handheld controller within the interactive contentobject and allowing interactions with the interactive content object viathe handheld controller; monitoring changes of the pointing vector withreference to the interactive content object; in response to detectingmovement of the pointing vector outside the interactive content object,updating the system to a second interaction mode wherein the handheldcontroller causes movement of the interactive content object such thatthe interactive content object follows movements of the handheldcontroller in the virtual environment; and in response to detectingmovement of the pointing vector of the handheld controller inside theinteractive content object, updating the system to the first interactionmode.

In some configurations, an augmented reality (AR) system can include: anAR display configured to present virtual content to a user of the ARsystem; a handheld controller having at least six degrees of freedom;and a hardware processor in communication with the AR display, theoutward facing camera, and the handheld controller, the hardwareprocessor can be programmed to: display interactive content at a firstcontent location; determine a first pointing vector comprising adirection indicated by the controller; determine whether the firstpointing vector intersects with a bounded volume associated with theinteractive content; in response to determining that the first pointingvector does not intersect the bounded volume, move the interactivecontent to a second content location associated with a point along thedirection of the first pointing vector; and in response to determiningthat the first pointing vector intersects the bounded volume, receive anindication to interact with the interactive content at the first contentlocation.

A method for displaying virtual content can include: displayinginteractive content at a first content location; determining a firstpointing vector comprising a direction indicated by a controller;determining whether the first pointing vector intersects with a boundedvolume associated with the interactive content; in response todetermining that the first pointing vector does not intersect thebounded volume, moving the interactive content to a second contentlocation associated with a point along the direction of the firstpointing vector; and in response to determining that the first pointingvector intersects the bounded volume, receiving an indication tointeract with the interactive content at the first content location.

BRIEF DESCRIPTION OF THE DRAWINGS

Throughout the drawings, reference numbers are re-used to indicatecorrespondence between referenced elements. The following drawings andthe associated descriptions are provided to illustrate embodiments ofthe present disclosure and do not limit the scope of the claims.

The drawings illustrate the design and utility of various embodiments ofthe present disclosure. It should be noted that the figures are notdrawn to scale and that elements of similar structures or functions arerepresented by like reference numerals throughout the figures. In orderto better appreciate how to obtain the above-recited and otheradvantages and objects of various embodiments of the disclosure, a moredetailed description of the present disclosure briefly described abovewill be rendered by reference to specific embodiments thereof, which areillustrated in the accompanying drawings. Understanding that thesedrawings depict only typical embodiments of the disclosure and are nottherefore to be considered limiting of its scope, the disclosure will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 illustrates a user’s view of an augmented reality (AR) scenethrough an AR device.

FIG. 2 illustrates a conventional display system for simulatingthree-dimensional imagery for a user.

FIGS. 3A-3C illustrate relationships between radius of curvature andfocal radius.

FIG. 4A illustrates a representation of the accommodation-vergenceresponse of the human visual system.

FIG. 4B illustrates examples of different accommodative states andvergence states of a pair of eyes of the user.

FIG. 4C illustrates an example of a representation of a top-down view ofa user viewing content via a display system.

FIG. 4D illustrates another example of a representation of a top-downview of a user viewing content via a display system.

FIG. 5 illustrates aspects of an approach for simulatingthree-dimensional imagery by modifying wavefront divergence.

FIG. 6 illustrates an example of a waveguide stack for outputting imageinformation to a user.

FIG. 7 illustrates an example of exit beams outputted by a waveguide.

FIG. 8 illustrates an example of a stacked waveguide assembly in whicheach depth plane includes images formed using multiple differentcomponent colors.

FIG. 9A illustrates a cross-sectional side view of an example of a setof stacked waveguides that each includes an in-coupling optical element.

FIG. 9B illustrates a perspective view of an example of the plurality ofstacked waveguides of FIG. 9A.

FIG. 9C illustrates a top-down plan view of an example of the pluralityof stacked waveguides of FIGS. 9A and 9B.

FIG. 9D illustrates an example of wearable display system.

FIG. 10A illustrates examples of user inputs received through controllerbuttons.

FIG. 10B illustrates examples of user inputs received through acontroller touchpad.

FIG. 10C illustrates examples of user inputs received through physicalmovement of a controller or a head-mounted device (HMD).

FIG. 10D illustrates examples of how user inputs may have differentdurations.

FIG. 11A illustrates additional examples of user inputs received throughcontroller buttons.

FIG. 11B illustrates additional examples of user inputs received througha controller touchpad.

FIG. 12A illustrates an example content control environment 1200according to one example of a follow and interaction method disclosedherein.

FIG. 12B shows a flowchart of an example process that may be utilized tocontrol content location in the user’s 3D space or interact with thecontent.

FIG. 12C illustrates example content within a bounded volume.

FIG. 12D illustrates a top view of a controller oriented such that itspointing vector is generally towards the content within a boundedvolume.

FIGS. 13A, 13B and 13C illustrate example aspects of an example contentmovement environment.

FIGS. 14A-14C illustrate aspects of an example content orientationenvironment.

FIG. 14D illustrates a flow chart of an example content orientationprocess.

FIG. 15 illustrates an example content access mechanic.

FIG. 16 illustrates an example content drop mechanic.

FIG. 17 depicts an example application for a content follow system wheretwo users of respective wearable systems are conducting a telepresencesession.

DETAILED DESCRIPTION

AR and/or VR systems may display virtual content to a user, or viewer.For example, this content may be displayed on a head-mounted display,e.g., as part of eyewear, that projects image information to the user’seyes. In addition, where the system is an AR system, the display mayalso transmit light from the surrounding environment to the user’s eyes,to allow a view of that surrounding environment. As used herein, it willbe appreciated that a “head-mounted” or “head mountable” display is adisplay that may be mounted on the head of a viewer or user. Suchdisplays may be understood to form parts of a display system.

In various augmented reality and virtual reality display systems, anout-coupling element such as a diffractive or reflective out-couplingelement out-couples light from a waveguide toward an eye of a user.However, a typical out-coupling element may not be directional, and maybe structured so as to output light in a variety of directions at allregions within the out-coupling optical element. Thus, while a portionof the light out-coupled by the out-coupling element may be usefullydirected toward the pupil of the eye of the user where it will inter theeye to form images, other light out-coupled by the out-coupling elementmay not be incident at or near the pupil of the eye and thus does notcontribute to the images formed by the display system.

Accordingly, it may be desirable to design display systems that canincrease or optimize efficiency by targeting the out-coupling of lighttoward the pupil of the eye of the user, and by reducing the amount oflight that is out-coupled in other directions. By targeting out-couplingtoward the pupil, such systems may reduce the amount of light energythat must be generated by the light projection system or other displaylight source in order to produce an image of a given brightness in theeye of a wearer. Various embodiments of the present technology providesystems including out-coupling elements, in-coupling elements, and/orlight projection systems configured to selectively direct image lighttoward the pupil of the eye of the wearer. Such systems may therebyadvantageously improve the efficiency of the display devices disclosedherein, as they may increase the proportion of a given amount of lightproduced by a light projection system that is used to form imagesperceived by a user, and may reduce the proportion of the light thatfalls on other portions of the user’s eye or face, or that otherwisedoes not contribute to the images perceived by the user.

Reference will now be made to the drawings, in which like referencenumerals refer to like parts throughout. Unless indicated otherwise, thedrawings are schematic and not necessarily drawn to scale.

A. Terms

To facilitate an understanding of the systems and methods discussedherein, a number of terms are described below. The terms describedbelow, as well as other terms used herein, should be construed toinclude the provided descriptions, the ordinary and customary meaning ofthe terms, and/or any other implied meaning for the respective terms,wherein such construction is consistent with context of the term. Thus,the descriptions below do not limit the meaning of these terms, but onlyprovide example descriptions.

Headpose (or “head pose”): position and/or orientation of a wearableheadset and, by proxy, of the user’s head in the real world. Head posemay be determined using sensors such as inertial measurement units(IMUs), accelerometers, gyroscopes, etc. A head pose ray that extends inthe direction of the head pose may be used to interact with virtualobjects. For example, when a user is pointing or looking at a prism oran object, the object or prism is intersected by the user’s head poseray.

Controller: a handheld controller, such as a totem. A control mayprovide multiple degrees of freedom movement, such as 6DoF (Six Degreesof Freedom).

Controller pose: position and/or orientation of a controller. Acontroller pose can be used to determine an area, volume, or point of amixed reality environment at which the controller is pointing.

Prism: container, area, or volume associated with mixed reality contentor a mixed reality space. For example, a prism may contain one or morevirtual content items that may be selectable by a user. A prism mayspawn when an application is launched, and then may spawn sibling orchild prisms to create flexible layouts. An application within a prismmay be configured to control where these hierarchical prisms willappear, which is typically within the proximity of the first prism andeasily discoverable by a user. A prism may provide feedback to a user.In some embodiments, the feedback may be a title that is only displayedto the user when the prism is targeted with head pose. In someembodiments, the feedback may be a glow around the prism. Prism glow(and/or other prism feedback) may also be used in sharing to give userfeedback of which prisms are being shared.

Focus: characteristic of an object, such as a prism, that allowsinteractive objects to be selected.

Input Focus: characteristic of an object, such as a prism or applicationthat causes the object’s cursor to be refreshed and rendered as theactive system cursor. In some implementations, there can be multiplefocus objects but only one with input focus.

3D Content: A virtual object that can be displayed in the 3D environmentof the user. The 3D content can be static, animated, manipulable, orotherwise interacted with. 3D content can include web based contentgenerated by user interactions with web domains or web pages using, forexample, the browser tile.

A. Example Augmented Reality Scenario

FIG. 1 depicts an illustration of an augmented reality scenario withcertain virtual reality objects, and certain actual reality objectsviewed by a person. FIG. 1 depicts an augmented reality scene 100,wherein a user of an AR technology sees a real-world park-like setting110 featuring people, trees, buildings in the background, and a concreteplatform 120. In addition to these items, the user of the AR technologyalso perceives that he “sees” a robot statue 130 standing upon thereal-world platform 120, and a cartoon-like avatar character 140 (e.g.,a bumble bee) flying by which seems to be a personification of a bumblebee, even though these elements do not exist in the real world.

In order for a three-dimensional (3-D) display to produce a truesensation of depth, and more specifically, a simulated sensation ofsurface depth, it is desirable for each point in the display’s visualfield to generate the accommodative response corresponding to itsvirtual depth. If the accommodative response to a display point does notcorrespond to the virtual depth of that point, as determined by thebinocular depth cues of convergence and stereopsis, the human eye mayexperience an accommodation conflict, resulting in unstable imaging,harmful eye strain, headaches, and, in the absence of accommodationinformation, almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer’s eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane and/or based on observingdifferent image features on different depth planes being out of focus.As discussed elsewhere herein, such depth cues provide credibleperceptions of depth.

B. Example Display Systems

FIG. 2 illustrates a conventional display system for simulatingthree-dimensional imagery for a user. It will be appreciated that auser’s eyes are spaced apart and that, when looking at a real object inspace, each eye will have a slightly different view of the object andmay form an image of the object at different locations on the retina ofeach eye. This may be referred to as binocular disparity and may beutilized by the human visual system to provide a perception of depth.Conventional display systems simulate binocular disparity by presentingtwo distinct images 190, 200 with slightly different views of the samevirtual object—one for each eye 210, 220—corresponding to the views ofthe virtual object that would be seen by each eye were the virtualobject a real object at a desired depth. These images provide binocularcues that the user’s visual system may interpret to derive a perceptionof depth.

With continued reference to FIG. 2 , the images 190, 200 are spaced fromthe eyes 210, 220 by a distance 230 on a z-axis. The z-axis is parallelto the optical axis of the viewer with their eyes fixated on an objectat optical infinity directly ahead of the viewer. The images 190, 200are flat and at a fixed distance from the eyes 210, 220. Based on theslightly different views of a virtual object in the images presented tothe eyes 210, 220, respectively, the eyes may naturally rotate such thatan image of the object falls on corresponding points on the retinas ofeach of the eyes, to maintain single binocular vision. This rotation maycause the lines of sight of each of the eyes 210, 220 to converge onto apoint in space at which the virtual object is perceived to be present.As a result, providing three-dimensional imagery conventionally involvesproviding binocular cues that may manipulate the vergence of the user’seyes 210, 220, and that the human visual system interprets to provide aperception of depth.

Generating a realistic and comfortable perception of depth ischallenging, however. It will be appreciated that light from objects atdifferent distances from the eyes have wavefronts with different amountsof divergence. FIGS. 3A-3C illustrate relationships between distance andthe divergence of light rays. The distance between the object and theeye 210 is represented by, in order of decreasing distance, R1, R2, andR3. As shown in FIGS. 3A-3C, the light rays become more divergent asdistance to the object decreases. Conversely, as distance increases, thelight rays become more collimated. Stated another way, it will be saidthat the light field produced by a point (the object or a part of theobject) has a spherical wavefront curvature, which is a function of howfar away the point is from the eye of the user. The curvature increaseswith decreasing distance between the object and the eye 210. While onlya single eye 210 is illustrated for clarity of illustration in FIGS.3A-3C and other figures herein, the discussions regarding eye 210 may beapplied to both eyes 210 and 220 of a viewer.

With continued reference to FIGS. 3A-3C, light from an object that theviewer’s eyes are fixated on may have different degrees of wavefrontdivergence. Due to the different amounts of wavefront divergence, thelight may be focused differently by the lens of the eye, which in turnmay require the lens to assume different shapes to form a focused imageon the retina of the eye. Where a focused image is not formed on theretina, the resulting retinal blur acts as a cue to accommodation thatcauses a change in the shape of the lens of the eye until a focusedimage is formed on the retina. For example, the cue to accommodation maytrigger the ciliary muscles surrounding the lens of the eye to relax orcontract, thereby modulating the force applied to the suspensoryligaments holding the lens, thus causing the shape of the lens of theeye to change until retinal blur of an object of fixation is eliminatedor minimized, thereby forming a focused image of the object of fixationon the retina (e.g., fovea) of the eye. The process by which the lens ofthe eye changes shape may be referred to as accommodation, and the shapeof the lens of the eye required to form a focused image of the object offixation on the retina (e.g., fovea) of the eye may be referred to as anaccommodative state.

With reference now to FIG. 4A, a representation of theaccommodation-vergence response of the human visual system isillustrated. The movement of the eyes to fixate on an object causes theeyes to receive light from the object, with the light forming an imageon each of the retinas of the eyes. The presence of retinal blur in theimage formed on the retina may provide a cue to accommodation, and therelative locations of the image on the retinas may provide a cue tovergence. The cue to accommodation causes accommodation to occur,resulting in the lenses of the eyes each assuming a particularaccommodative state that forms a focused image of the object on theretina (e.g., fovea) of the eye. On the other hand, the cue to vergencecauses vergence movements (rotation of the eyes) to occur such that theimages formed on each retina of each eye are at corresponding retinalpoints that maintain single binocular vision. In these positions, theeyes may be said to have assumed a particular vergence state. Withcontinued reference to FIG. 4A, accommodation may be understood to bethe process by which the eye achieves a particular accommodative state,and vergence may be understood to be the process by which the eyeachieves a particular vergence state. As indicated in FIG. 4A, theaccommodative and vergence states of the eyes may change if the userfixates on another object. For example, the accommodated state maychange if the user fixates on a new object at a different depth on thez-axis.

Without being limited by theory, it is believed that viewers of anobject may perceive the object as being “three-dimensional” due to acombination of vergence and accommodation. As noted above, vergencemovements (e.g., rotation of the eyes so that the pupils move toward oraway from each other to converge the lines of sight of the eyes tofixate upon an object) of the two eyes relative to each other areclosely associated with accommodation of the lenses of the eyes. Undernormal conditions, changing the shapes of the lenses of the eyes tochange focus from one object to another object at a different distancewill automatically cause a matching change in vergence to the samedistance, under a relationship known as the “accommodation-vergencereflex.” Likewise, a change in vergence will trigger a matching changein lens shape under normal conditions.

With reference now to FIG. 4B, examples of different accommodative andvergence states of the eyes are illustrated. The pair of eyes 222 a isfixated on an object at optical infinity, while the pair eyes 222 b arefixated on an object 221 at less than optical infinity. Notably, thevergence states of each pair of eyes is different, with the pair of eyes222 a directed straight ahead, while the pair of eyes 222 converge onthe object 221. The accommodative states of the eyes forming each pairof eyes 222 a and 222 b are also different, as represented by thedifferent shapes of the lenses 210 a, 220 a.

Undesirably, many users of conventional “3-D” display systems find suchconventional systems to be uncomfortable or may not perceive a sense ofdepth at all due to a mismatch between accommodative and vergence statesin these displays. As noted above, many stereoscopic or “3-D” displaysystems display a scene by providing slightly different images to eacheye. Such systems are uncomfortable for many viewers, since they, amongother things, simply provide different presentations of a scene andcause changes in the vergence states of the eyes, but without acorresponding change in the accommodative states of those eyes. Rather,the images are shown by a display at a fixed distance from the eyes,such that the eyes view all the image information at a singleaccommodative state. Such an arrangement works against the“accommodation-vergence reflex” by causing changes in the vergence statewithout a matching change in the accommodative state. This mismatch isbelieved to cause viewer discomfort. Display systems that provide abetter match between accommodation and vergence may form more realisticand comfortable simulations of three-dimensional imagery.

Without being limited by theory, it is believed that the human eyetypically may interpret a finite number of depth planes to provide depthperception. Consequently, a highly believable simulation of perceiveddepth may be achieved by providing, to the eye, different presentationsof an image corresponding to each of these limited numbers of depthplanes. In some embodiments, the different presentations may provideboth cues to vergence and matching cues to accommodation, therebyproviding physiologically correct accommodation-vergence matching.

With continued reference to FIG. 4B, two depth planes 240, correspondingto different distances in space from the eyes 210, 220, are illustrated.For a given depth plane 240, vergence cues may be provided by thedisplaying of images of appropriately different perspectives for eacheye 210, 220. In addition, for a given depth plane 240, light formingthe images provided to each eye 210, 220 may have a wavefront divergencecorresponding to a light field produced by a point at the distance ofthat depth plane 240.

In the illustrated embodiment, the distance, along the z-axis, of thedepth plane 240 containing the point 221 is 1 m. As used herein,distances or depths along the z-axis may be measured with a zero-pointlocated at the exit pupils of the user’s eyes. Thus, a depth plane 240located at a depth of 1 m corresponds to a distance of 1 m away from theexit pupils of the user’s eyes, on the optical axis of those eyes withthe eyes directed towards optical infinity. As an approximation, thedepth or distance along the z-axis may be measured from the display infront of the user’s eyes (e.g., from the surface of a waveguide), plus avalue for the distance between the device and the exit pupils of theuser’s eyes. That value may be called the eye relief and corresponds tothe distance between the exit pupil of the user’s eye and the displayworn by the user in front of the eye. In practice, the value for the eyerelief may be a normalized value used generally for all viewers. Forexample, the eye relief may be assumed to be 20 mm and a depth planethat is at a depth of 1 m may be at a distance of 980 mm in front of thedisplay.

With reference now to FIGS. 4C and 4D, examples of matchedaccommodation-vergence distances and mismatched accommodation-vergencedistances are illustrated, respectively. As illustrated in FIG. 4C, thedisplay system may provide images of a virtual object to each eye 210,220. The images may cause the eyes 210, 220 to assume a vergence statein which the eyes converge on a point 15 on a depth plane 240. Inaddition, the images may be formed by a light having a wavefrontcurvature corresponding to real objects at that depth plane 240. As aresult, the eyes 210, 220 assume an accommodative state in which theimages are in focus on the retinas of those eyes. Thus, the user mayperceive the virtual object as being at the point 15 on the depth plane240.

It will be appreciated that each of the accommodative and vergencestates of the eyes 210, 220 are associated with a particular distance onthe z-axis. For example, an object at a particular distance from theeyes 210, 220 causes those eyes to assume particular accommodativestates based upon the distances of the object. The distance associatedwith a particular accommodative state may be referred to as theaccommodation distance, A_(d). Similarly, there are particular vergencedistances, V_(d), associated with the eyes in particular vergencestates, or positions relative to one another. Where the accommodationdistance and the vergence distance match, the relationship betweenaccommodation and vergence may be said to be physiologically correct.This is considered to be the most comfortable scenario for a viewer.

In stereoscopic displays, however, the accommodation distance and thevergence distance may not always match. For example, as illustrated inFIG. 4D, images displayed to the eyes 210, 220 may be displayed withwavefront divergence corresponding to depth plane 240, and the eyes 210,220 may assume a particular accommodative state in which the points 15a, 15 b on that depth plane are in focus. However, the images displayedto the eyes 210, 220 may provide cues for vergence that cause the eyes210, 220 to converge on a point 15 that is not located on the depthplane 240. As a result, the accommodation distance corresponds to thedistance from the exit pupils of the eyes 210, 220 to the depth plane240, while the vergence distance corresponds to the larger distance fromthe exit pupils of the eyes 210, 220 to the point 15, in someembodiments. The accommodation distance is different from the vergencedistance. Consequently, there is an accommodation-vergence mismatch.Such a mismatch is considered undesirable and may cause discomfort inthe user. It will be appreciated that the mismatch corresponds todistance (e.g., V_(d) - A_(d)) and may be characterized using diopters.

In some embodiments, it will be appreciated that a reference point otherthan exit pupils of the eyes 210, 220 may be utilized for determiningdistance for determining accommodation-vergence mismatch, so long as thesame reference point is utilized for the accommodation distance and thevergence distance. For example, the distances could be measured from thecornea to the depth plane, from the retina to the depth plane, from theeyepiece (e.g., a waveguide of the display device) to the depth plane,and so on.

Without being limited by theory, it is believed that users may stillperceive accommodation-vergence mismatches of up to about 0.25 diopter,up to about 0.33 diopter, and up to about 0.5 diopter as beingphysiologically correct, without the mismatch itself causing significantdiscomfort. In some embodiments, display systems disclosed herein (e.g.,the display system 250, FIG. 6 ) present images to the viewer havingaccommodation-vergence mismatch of about 0.5 diopter or less. In someother embodiments, the accommodation-vergence mismatch of the imagesprovided by the display system is about 0.33 diopter or less. In yetother embodiments, the accommodation-vergence mismatch of the imagesprovided by the display system is about 0.25 diopter or less, includingabout 0.1 diopter or less.

FIG. 5 illustrates aspects of an approach for simulatingthree-dimensional imagery by modifying wavefront divergence. The displaysystem includes a waveguide 270 that is configured to receive light 770that is encoded with image information, and to output that light to theuser’s eye 210. The waveguide 270 may output the light 650 with adefined amount of wavefront divergence corresponding to the wavefrontdivergence of a light field produced by a point on a desired depth plane240. In some embodiments, the same amount of wavefront divergence isprovided for all objects presented on that depth plane. In addition, itwill be illustrated that the other eye of the user may be provided withimage information from a similar waveguide.

In some embodiments, a single waveguide may be configured to outputlight with a set amount of wavefront divergence corresponding to asingle or limited number of depth planes and/or the waveguide may beconfigured to output light of a limited range of wavelengths.Consequently, in some embodiments, a plurality or stack of waveguidesmay be utilized to provide different amounts of wavefront divergence fordifferent depth planes and/or to output light of different ranges ofwavelengths. As used herein, it will be appreciated at a depth plane maybe planar or may follow the contours of a curved surface.

FIG. 6 illustrates an example of a waveguide stack for outputting imageinformation to a user. A display system 250 includes a stack ofwaveguides, or stacked waveguide assembly, 260 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 270, 280, 290, 300, 310. It will be appreciated that thedisplay system 250 may be considered a light field display in someembodiments. In addition, the waveguide assembly 260 may also bereferred to as an eyepiece.

In some embodiments, the display system 250 may be configured to providesubstantially continuous cues to vergence and multiple discrete cues toaccommodation. The cues to vergence may be provided by displayingdifferent images to each of the eyes of the user, and the cues toaccommodation may be provided by outputting the light that forms theimages with selectable discrete amounts of wavefront divergence. Statedanother way, the display system 250 may be configured to output lightwith variable levels of wavefront divergence. In some embodiments, eachdiscrete level of wavefront divergence corresponds to a particular depthplane and may be provided by a particular one of the waveguides 270,280, 290, 300, 310.

With continued reference to FIG. 6 , the waveguide assembly 260 may alsoinclude a plurality of features 320, 330, 340, 350 between thewaveguides. In some embodiments, the features 320, 330, 340, 350 may beone or more lenses. The waveguides 270, 280, 290, 300, 310 and/or theplurality of lenses 320, 330, 340, 350 may be configured to send imageinformation to the eye with various levels of wavefront curvature orlight ray divergence. Each waveguide level may be associated with aparticular depth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 360, 370,380, 390, 400 may function as a source of light for the waveguides andmay be utilized to inject image information into the waveguides 270,280, 290, 300, 310, each of which may be configured, as describedherein, to distribute incoming light across each respective waveguide,for output toward the eye 210. Light exits an output surface 410, 420,430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 andis injected into a corresponding input surface 460, 470, 480, 490, 500of the waveguides 270, 280, 290, 300, 310. In some embodiments, each ofthe input surfaces 460, 470, 480, 490, 500 may be an edge of acorresponding waveguide, or may be part of a major surface of thecorresponding waveguide (that is, one of the waveguide surfaces directlyfacing the world 510 or the viewer’s eye 210). In some embodiments, asingle beam of light (e.g., a collimated beam) may be injected into eachwaveguide to output an entire field of cloned collimated beams that aredirected toward the eye 210 at particular angles (and amounts ofdivergence) corresponding to the depth plane associated with aparticular waveguide. In some embodiments, a single one of the imageinjection devices 360, 370, 380, 390, 400 may be associated with andinject light into a plurality (e.g., three) of the waveguides 270, 280,290, 300, 310.

In some embodiments, the image injection devices 360, 370, 380, 390, 400are discrete displays that each produce image information for injectioninto a corresponding waveguide 270, 280, 290, 300, 310, respectively. Insome other embodiments, the image injection devices 360, 370, 380, 390,400 are the output ends of a single multiplexed display which may, e.g.,pipe image information via one or more optical conduits (such as fiberoptic cables) to each of the image injection devices 360, 370, 380, 390,400. It will be appreciated that the image information provided by theimage injection devices 360, 370, 380, 390, 400 may include light ofdifferent wavelengths, or colors (e.g., different component colors, asdiscussed herein).

In some embodiments, the light injected into the waveguides 270, 280,290, 300, 310 is encoded with image information and provided by a lightprojector system 1010, as discussed further herein. In some embodiments,the light projector system 1010 may comprise one or more emissive pixelarrays. It will be appreciated that the emissive pixel arrays may eachcomprise a plurality of light-emitting pixels, which may be configuredto emit light of varying intensities and colors. It will be appreciatedthat the image injection devices 360, 370, 380, 390, 400 are illustratedschematically and, in some embodiments, these image injection devicesmay represent different light paths and locations in a common projectionsystem configured to output light into associated ones of the waveguides270, 280, 290, 300, 310. In some embodiments, the waveguides of thewaveguide assembly 260 may function as ideal lens while relaying lightinjected into the waveguides out to the user’s eyes. In this conception,the object may be the pixel array of the light projector system 1010 andthe image may be the image on the depth plane.

A controller 560 controls the operation of one or more of the stackedwaveguide assembly 260, including operation of the image injectiondevices 360, 370, 380, 390, 400, the light projection system 1010. Insome embodiments, the controller 560 is part of the local dataprocessing module 140. The controller 560 includes programming (e.g.,instructions in a non-transitory medium) that regulates the timing andprovision of image information to the waveguides 270, 280, 290, 300, 310according to, e.g., any of the various schemes disclosed herein. In someembodiments, the controller may be a single integral device, or adistributed system connected by wired or wireless communicationchannels. The controller 560 may be part of the processing modules 140or 150 (FIG. 9D) in some embodiments.

With continued reference to FIG. 6 , the waveguides 270, 280, 290, 300,310 may be configured to propagate light within each respectivewaveguide by total internal reflection (TIR). The waveguides 270, 280,290, 300, 310 may each be planar or have another shape (e.g., curved),with major top and bottom surfaces and edges extending between thosemajor top and bottom surfaces. In the illustrated configuration, thewaveguides 270, 280, 290, 300, 310 may each include out-coupling opticalelements 570, 580, 590, 600, 610 that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 210. Extracted light may also be referred to as out-coupledlight and the out-coupling optical elements light may also be referredto light extracting optical elements. An extracted beam of light may beoutputted by the waveguide at locations at which the light propagatingin the waveguide strikes a light extracting optical element. Theout-coupling optical elements 570, 580, 590, 600, 610 may, for example,be gratings, including diffractive optical features, as discussedfurther herein. While illustrated disposed at the bottom major surfacesof the waveguides 270, 280, 290, 300, 310, for ease of description anddrawing clarity, in some embodiments, the out-coupling optical elements570, 580, 590, 600, 610 may be disposed at the top and/or bottom majorsurfaces, and/or may be disposed directly in the volume of thewaveguides 270, 280, 290, 300, 310, as discussed further herein. In someembodiments, the out-coupling optical elements 570, 580, 590, 600, 610may be formed in a layer of material that is attached to a transparentsubstrate to form the waveguides 270, 280, 290, 300, 310. In some otherembodiments, the waveguides 270, 280, 290, 300, 310 may be a monolithicpiece of material and the out-coupling optical elements 570, 580, 590,600, 610 may be formed on a surface and/or in the interior of that pieceof material.

With continued reference to FIG. 6 , as discussed herein, each waveguide270, 280, 290, 300, 310 is configured to output light to form an imagecorresponding to a particular depth plane. For example, the waveguide270 nearest the eye may be configured to deliver collimated light (whichwas injected into such waveguide 270), to the eye 210. The collimatedlight may be representative of the optical infinity focal plane. Thenext waveguide up 280 may be configured to send out collimated lightwhich passes through the first lens 350 (e.g., a negative lens) beforeit may reach the eye 210; such first lens 350 may be configured tocreate a slight convex wavefront curvature so that the eye/braininterprets light coming from that next waveguide up 280 as coming from afirst focal plane closer inward toward the eye 210 from opticalinfinity. Similarly, the third up waveguide 290 passes its output lightthrough both the first 350 and second 340 lenses before reaching the eye210; the combined optical power of the first 350 and second 340 lensesmay be configured to create another incremental amount of wavefrontcurvature so that the eye/brain interprets light coming from the thirdwaveguide 290 as coming from a second focal plane that is even closerinward toward the person from optical infinity than was light from thenext waveguide up 280.

The other waveguide layers 300, 310 and lenses 330, 320 are similarlyconfigured, with the highest waveguide 310 in the stack sending itsoutput through all of the lenses between it and the eye for an aggregatefocal power representative of the closest focal plane to the person. Tocompensate for the stack of lenses 320, 330, 340, 350 whenviewing/interpreting light coming from the world 510 on the other sideof the stacked waveguide assembly 260, a compensating lens layer 620 maybe disposed at the top of the stack to compensate for the aggregatepower of the lens stack 320, 330, 340, 350 below. Such a configurationprovides as many perceived focal planes as there are availablewaveguide/lens pairings. Both the out-coupling optical elements of thewaveguides and the focusing aspects of the lenses may be static (i.e.,not dynamic or electro-active). In some alternative embodiments, eitheror both may be dynamic using electro-active features.

In some embodiments, two or more of the waveguides 270, 280, 290, 300,310 may have the same associated depth plane. For example, multiplewaveguides 270, 280, 290, 300, 310 may be configured to output imagesset to the same depth plane, or multiple subsets of the waveguides 270,280, 290, 300, 310 may be configured to output images set to the sameplurality of depth planes, with one set for each depth plane. This mayprovide advantages for forming a tiled image to provide an expandedfield of view at those depth planes.

With continued reference to FIG. 6 , the out-coupling optical elements570, 580, 590, 600, 610 may be configured to both redirect light out oftheir respective waveguides and to output this light with theappropriate amount of divergence or collimation for a particular depthplane associated with the waveguide. As a result, waveguides havingdifferent associated depth planes may have different configurations ofout-coupling optical elements 570, 580, 590, 600, 610, which outputlight with a different amount of divergence depending on the associateddepth plane. In some embodiments, the light extracting optical elements570, 580, 590, 600, 610 may be volumetric or surface features, which maybe configured to output light at specific angles. For example, the lightextracting optical elements 570, 580, 590, 600, 610 may be volumeholograms, surface holograms, and/or diffraction gratings. In someembodiments, the features 320, 330, 340, 350 may not be lenses; rather,they may simply be spacers (e.g., cladding layers and/or structures forforming air gaps).

In some embodiments, the out-coupling optical elements 570, 580, 590,600, 610 are diffractive features that form a diffraction pattern, or“diffractive optical element” (also referred to herein as a “DOE”).Preferably, the DOE’s have a sufficiently low diffraction efficiency sothat only a portion of the light of the beam is deflected away towardthe eye 210 with each intersection of the DOE, while the rest continuesto move through a waveguide via TIR. The light carrying the imageinformation is thus divided into a number of related exit beams thatexit the waveguide at a multiplicity of locations and the result is afairly uniform pattern of exit emission toward the eye 210 for thisparticular collimated beam bouncing around within a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”states in which they actively diffract, and “off” states in which theydo not significantly diffract. For instance, a switchable DOE maycomprise a layer of polymer dispersed liquid crystal, in whichmicrodroplets comprise a diffraction pattern in a host medium, and therefractive index of the microdroplets may be switched to substantiallymatch the refractive index of the host material (in which case thepattern does not appreciably diffract incident light) or themicrodroplet may be switched to an index that does not match that of thehost medium (in which case the pattern actively diffracts incidentlight).

In some embodiments, a camera assembly 630 (e.g., a digital camera,including visible light and infrared light cameras) may be provided tocapture images of the eye 210 and/or tissue around the eye 210 to, e.g.,detect user inputs and/or to monitor the physiological state of theuser. As used herein, a camera may be any image capture device. In someembodiments, the camera assembly 630 may include an image capture deviceand a light source to project light (e.g., infrared light) to the eye,which may then be reflected by the eye and detected by the image capturedevice. In some embodiments, the camera assembly 630 may be attached tothe frame 80 (FIG. 9D) and may be in electrical communication with theprocessing modules 140 and/or 150, which may process image informationfrom the camera assembly 630. In some embodiments, one camera assembly630 may be utilized for each eye, to separately monitor each eye.

With reference now to FIG. 7 , an example of exit beams outputted by awaveguide is shown. One waveguide is illustrated, but it will beappreciated that other waveguides in the waveguide assembly 260 (FIG. 6) may function similarly, where the waveguide assembly 260 includesmultiple waveguides. Light 640 is injected into the waveguide 270 at theinput surface 460 of the waveguide 270 and propagates within thewaveguide 270 by TIR. At points where the light 640 impinges on the DOE570, a portion of the light exits the waveguide as exit beams 650. Theexit beams 650 are illustrated as substantially parallel but, asdiscussed herein, they may also be redirected to propagate to the eye210 at an angle (e.g., forming divergent exit beams), depending on thedepth plane associated with the waveguide 270. It will be appreciatedthat substantially parallel exit beams may be indicative of a waveguidewith out-coupling optical elements that out-couple light to form imagesthat appear to be set on a depth plane at a large distance (e.g.,optical infinity) from the eye 210. Other waveguides or other sets ofout-coupling optical elements may output an exit beam pattern that ismore divergent, which would require the eye 210 to accommodate to acloser distance to bring it into focus on the retina and would beinterpreted by the brain as light from a distance closer to the eye 210than optical infinity.

In some embodiments, a full color image may be formed at each depthplane by overlaying images in each of the component colors, e.g., threeor more component colors. FIG. 8 illustrates an example of a stackedwaveguide assembly in which each depth plane includes images formedusing multiple different component colors. The illustrated embodimentshows depth planes 240 a - 240 f, although more or fewer depths are alsocontemplated. Each depth plane may have three or more component colorimages associated with it, including: a first image of a first color, G;a second image of a second color, R; and a third image of a third color,B. Different depth planes are indicated in the figure by differentnumbers for diopters (dpt) following the letters G, R, and B. Just asexamples, the numbers following each of these letters indicate diopters(1/m), or inverse distance of the depth plane from a viewer, and eachbox in the figures represents an individual component color image. Insome embodiments, to account for differences in the eye’s focusing oflight of different wavelengths, the exact placement of the depth planesfor different component colors may vary. For example, differentcomponent color images for a given depth plane may be placed on depthplanes corresponding to different distances from the user. Such anarrangement may increase visual acuity and user comfort and/or maydecrease chromatic aberrations.

In some embodiments, light of each component color may be outputted by asingle dedicated waveguide and, consequently, each depth plane may havemultiple waveguides associated with it. In such embodiments, each box inthe figures including the letters G, R, or B may be understood torepresent an individual waveguide, and three waveguides may be providedper depth plane where three component color images are provided perdepth plane. While the waveguides associated with each depth plane areshown adjacent to one another in this drawing for ease of description,it will be appreciated that, in a physical device, the waveguides mayall be arranged in a stack with one waveguide per level. In some otherembodiments, multiple component colors may be outputted by the samewaveguide, such that, e.g., only a single waveguide may be provided perdepth plane.

With continued reference to FIG. 8 , in some embodiments, G is the colorgreen, R is the color red, and B is the color blue. In some otherembodiments, other colors associated with other wavelengths of light,including magenta and cyan, may be used in addition to or may replaceone or more of red, green, or blue.

It will be appreciated that references to a given color of lightthroughout this disclosure will be understood to encompass light of oneor more wavelengths within a range of wavelengths of light that areperceived by a viewer as being of that given color. For example, redlight may include light of one or more wavelengths in the range of about620-780 nm, green light may include light of one or more wavelengths inthe range of about 492-577 nm, and blue light may include light of oneor more wavelengths in the range of about 435-493 nm.

In some embodiments, the light projection system 1010 (FIG. 6 ) may beconfigured to emit light of one or more wavelengths outside the visualperception range of the viewer, for example, infrared and/or ultravioletwavelengths. In addition, the in-coupling, out-coupling, and other lightredirecting structures of the waveguides of the display 250 may beconfigured to direct and emit this light out of the display towards theuser’s eye 210, e.g., for imaging and/or user stimulation applications.

With reference now to FIG. 9A, in some embodiments, light impinging on awaveguide may need to be redirected to in-couple that light into thewaveguide. An in-coupling optical element may be used to redirect andin-couple the light into its corresponding waveguide. FIG. 9Aillustrates a cross-sectional side view of an example of a plurality orset 660 of stacked waveguides that each includes an in-coupling opticalelement. The waveguides may each be configured to output light of one ormore different wavelengths, or one or more different ranges ofwavelengths. It will be appreciated that the stack 660 may correspond tothe stack 260 (FIG. 6 ) and the illustrated waveguides of the stack 660may correspond to part of the plurality of waveguides 270, 280, 290,300, 310, except that light from one or more of the image injectiondevices 360, 370, 380, 390, 400 is injected into the waveguides from aposition that requires light to be redirected for in-coupling.

The illustrated set 660 of stacked waveguides includes waveguides 670,680, and 690. Each waveguide includes an associated in-coupling opticalelement (which may also be referred to as a light input area on thewaveguide), with, e.g., in-coupling optical element 700 disposed on amajor surface (e.g., an upper major surface) of waveguide 670,incoupling optical element 710 disposed on a major surface (e.g., anupper major surface) of waveguide 680, and in-coupling optical element720 disposed on a major surface (e.g., an upper major surface) ofwaveguide 690. In some embodiments, one or more of the in-couplingoptical elements 700, 710, 720 may be disposed on the bottom majorsurface of the respective waveguide 670, 680, 690 (particularly wherethe one or more in-coupling optical elements are reflective, deflectingoptical elements). As illustrated, the in-coupling optical elements 700,710, 720 may be disposed on the upper major surface of their respectivewaveguide 670, 680, 690 (or the top of the next lower waveguide),particularly where those in-coupling optical elements are transmissive,deflecting optical elements. In some embodiments, the in-couplingoptical elements 700, 710, 720 may be disposed in the body of therespective waveguide 670, 680, 690. In some embodiments, as discussedherein, the in-coupling optical elements 700, 710, 720 are wavelengthselective, such that they selectively redirect one or more wavelengthsof light, while transmitting other wavelengths of light. Whileillustrated on one side or corner of their respective waveguide 670,680, 690, it will be appreciated that the in-coupling optical elements700, 710, 720 may be disposed in other areas of their respectivewaveguide 670, 680, 690 in some embodiments.

As illustrated, the in-coupling optical elements 700, 710, 720 may belaterally offset from one another. In some embodiments, each in-couplingoptical element may be offset such that it receives light without thatlight passing through another in-coupling optical element. For example,each in-coupling optical element 700, 710, 720 may be configured toreceive light from a different image injection device 360, 370, 380,390, and 400 as shown in FIG. 6 , and may be separated (e.g., laterallyspaced apart) from other in-coupling optical elements 700, 710, 720 suchthat it substantially does not receive light from the other ones of thein-coupling optical elements 700, 710, 720.

Each waveguide also includes associated light distributing elements,with, e.g., light distributing elements 730 disposed on a major surface(e.g., a top major surface) of waveguide 670, light distributingelements 740 disposed on a major surface (e.g., a top major surface) ofwaveguide 680, and light distributing elements 750 disposed on a majorsurface (e.g., a top major surface) of waveguide 690. In some otherembodiments, the light distributing elements 730, 740, 750, may bedisposed on a bottom major surface of associated waveguides 670, 680,690, respectively. In some other embodiments, the light distributingelements 730, 740, 750, may be disposed on both top and bottom majorsurface of associated waveguides 670, 680, 690, respectively; or thelight distributing elements 730, 740, 750, may be disposed on differentones of the top and bottom major surfaces in different associatedwaveguides 670, 680, 690, respectively.

The waveguides 670, 680, 690 may be spaced apart and separated by, e.g.,gas, liquid, and/or solid layers of material. For example, asillustrated, layer 760 a may separate waveguides 670 and 680; and layer760 b may separate waveguides 680 and 690. In some embodiments, thelayers 760 a and 760 b are formed of low refractive index materials(that is, materials having a lower refractive index than the materialforming the immediately adjacent one of waveguides 670, 680, 690).Preferably, the refractive index of the material forming the layers 760a, 760 b is 0.05 or more, or 0.10 or less than the refractive index ofthe material forming the waveguides 670, 680, 690. Advantageously, thelower refractive index layers 760 a, 760 b may function as claddinglayers that facilitate total internal reflection (TIR) of light throughthe waveguides 670, 680, 690 (e.g., TIR between the top and bottom majorsurfaces of each waveguide). In some embodiments, the layers 760 a, 760b are formed of air. While not illustrated, it will be appreciated thatthe top and bottom of the illustrated set 660 of waveguides may includeimmediately neighboring cladding layers.

Preferably, for ease of manufacturing and other considerations, thematerial forming the waveguides 670, 680, 690 are similar or the same,and the material forming the layers 760 a, 760 b are similar or thesame. In some embodiments, the material forming the waveguides 670, 680,690 may be different between one or more waveguides, and/or the materialforming the layers 760 a, 760 b may be different, while still holding tothe various refractive index relationships noted above.

With continued reference to FIG. 9A, light rays 770, 780, 790 areincident on the set 660 of waveguides. It will be appreciated that thelight rays 770, 780, 790 may be injected into the waveguides 670, 680,690 by one or more image injection devices 360, 370, 380, 390, 400 (FIG.6 ).

In some embodiments, the light rays 770, 780, 790 are intended fordifferent waveguides (e.g., waveguides configured to output light withdifferent amounts of wavefront divergence, and/or configured to outputlight having different properties, such as different wavelengths orcolors). Thus, in some embodiments, the light rays 770, 780, 790 mayhave different properties, e.g., different wavelengths or differentranges of wavelengths, which may correspond to different colors. Thein-coupling optical elements 700, 710, 720 each deflect the incidentlight such that the light propagates through a respective one of thewaveguides 670, 680, 690 by TIR. In some embodiments, the in-couplingoptical elements 700, 710, 720 each selectively deflect one or moreparticular wavelengths of light, while transmitting other wavelengths toan underlying waveguide and associated in-coupling optical element.

For example, in-coupling optical element 700 may be configured todeflect ray 770, which has a first wavelength or range of wavelengths,while transmitting rays 780 and 790, which have different second andthird wavelengths or ranges of wavelengths, respectively. Thetransmitted ray 780 impinges on and is deflected by the in-couplingoptical element 710, which is configured to deflect light of a secondwavelength or range of wavelengths. The ray 790 is deflected by thein-coupling optical element 720, which is configured to selectivelydeflect light of third wavelength or range of wavelengths.

With continued reference to FIG. 9A, the deflected light rays 770, 780,790 are deflected so that they propagate through a correspondingwaveguide 670, 680, 690; that is, the in-coupling optical elements 700,710, 720 of each waveguide deflects light into that correspondingwaveguide 670, 680, 690 to in-couple light into that correspondingwaveguide. The light rays 770, 780, 790 are deflected at angles thatcause the light to propagate through the respective waveguide 670, 680,690 by TIR. The light rays 770, 780, 790 propagate through therespective waveguide 670, 680, 690 by TIR until impinging on thewaveguide’s corresponding light distributing elements 730, 740, 750.

With reference now to FIG. 9B, a perspective view of an example of theplurality of stacked waveguides of FIG. 9A is illustrated. As notedabove, the in-coupled light rays 770, 780, 790, are deflected by thein-coupling optical elements 700, 710, 720, respectively, and thenpropagate by TIR within the waveguides 670, 680, 690, respectively. Thelight rays 770, 780, 790 then impinge on the light distributing elements730, 740, 750, respectively. The light distributing elements 730, 740,750 deflect the light rays 770, 780, 790 so that they propagate towardsthe out-coupling optical elements 800, 810, 820, respectively.

In some embodiments, the light distributing elements 730, 740, 750 areorthogonal pupil expanders (OPE’s). In some embodiments, the OPE’sdeflect or distribute light to the out-coupling optical elements 800,810, 820 and, in some embodiments, may also increase the beam or spotsize of this light as it propagates to the out-coupling opticalelements. In some embodiments, the light distributing elements 730, 740,750 may be omitted and the in-coupling optical elements 700, 710, 720may be configured to deflect light directly to the out-coupling opticalelements 800, 810, 820. For example, with reference to FIG. 9A, thelight distributing elements 730, 740, 750 may be replaced without-coupling optical elements 800, 810, 820, respectively. In someembodiments, the out-coupling optical elements 800, 810, 820 are exitpupils (EP’s) or exit pupil expanders (EPE’s) that direct light in aviewer’s eye 210 (FIG. 7 ). It will be appreciated that the OPE’s may beconfigured to increase the dimensions of the eye box in at least oneaxis and the EPE’s may be to increase the eye box in an axis crossing,e.g., orthogonal to, the axis of the OPEs. For example, each OPE may beconfigured to redirect a portion of the light striking the OPE to an EPEof the same waveguide, while allowing the remaining portion of the lightto continue to propagate down the waveguide. Upon impinging on the OPEagain, another portion of the remaining light is redirected to the EPE,and the remaining portion of that portion continues to propagate furtherdown the waveguide, and so on. Similarly, upon striking the EPE, aportion of the impinging light is directed out of the waveguide towardsthe user, and a remaining portion of that light continues to propagatethrough the waveguide until it strikes the EP again, at which timeanother portion of the impinging light is directed out of the waveguide,and so on. Consequently, a single beam of in-coupled light may be“replicated” each time a portion of that light is redirected by an OPEor EPE, thereby forming a field of cloned beams of light, as shown inFIG. 6 . In some embodiments, the OPE and/or EPE may be configured tomodify a size of the beams of light.

Accordingly, with reference to FIGS. 9A and 9B, in some embodiments, theset 660 of waveguides includes waveguides 670, 680, 690; in-couplingoptical elements 700, 710, 720; light distributing elements (e.g.,OPE’s) 730, 740, 750; and out-coupling optical elements (e.g., EP’s)800, 810, 820 for each component color. The waveguides 670, 680, 690 maybe stacked with an air gap/cladding layer between each one. Thein-coupling optical elements 700, 710, 720 redirect or deflect incidentlight (with different in-coupling optical elements receiving light ofdifferent wavelengths) into its waveguide. The light then propagates atan angle which will result in TIR within the respective waveguide 670,680, 690. In the example shown, light ray 770 (e.g., blue light) isdeflected by the first in-coupling optical element 700, and thencontinues to bounce down the waveguide, interacting with the lightdistributing element (e.g., OPE’s) 730 and then the out-coupling opticalelement (e.g., EPs) 800, in a manner described earlier. The light rays780 and 790 (e.g., green and red light, respectively) will pass throughthe waveguide 670, with light ray 780 impinging on and being deflectedby in-coupling optical element 710. The light ray 780 then bounces downthe waveguide 680 via TIR, proceeding on to its light distributingelement (e.g., OPEs) 740 and then the out-coupling optical element(e.g., EP’s) 810. Finally, light ray 790 (e.g., red light) passesthrough the waveguide 690 to impinge on the light in-coupling opticalelements 720 of the waveguide 690. The light in-coupling opticalelements 720 deflect the light ray 790 such that the light raypropagates to light distributing element (e.g., OPEs) 750 by TIR, andthen to the out-coupling optical element (e.g., EPs) 820 by TIR. Theout-coupling optical element 820 then finally out-couples the light ray790 to the viewer, who also receives the out-coupled light from theother waveguides 670, 680.

FIG. 9C illustrates a top-down plan view of an example of the pluralityof stacked waveguides of FIGS. 9A and 9B. As illustrated, the waveguides670, 680, 690, along with each waveguide’s associated light distributingelement 730, 740, 750 and associated out-coupling optical element 800,810, 820, may be vertically aligned. However, as discussed herein, thein-coupling optical elements 700, 710, 720 are not vertically aligned;rather, the in-coupling optical elements are preferably non-overlapping(e.g., laterally spaced apart as seen in the top-down view). Asdiscussed further herein, this nonoverlapping spatial arrangementfacilitates the injection of light from different resources intodifferent waveguides on a one-to-one basis, thereby allowing a specificlight source to be uniquely coupled to a specific waveguide. In someembodiments, arrangements including nonoverlapping spatially-separatedin-coupling optical elements may be referred to as a shifted pupilsystem, and the in-coupling optical elements within these arrangementsmay correspond to sub pupils.

FIG. 9D illustrates an example of wearable display system 60 into whichthe various waveguides and related systems disclosed herein may beintegrated. In some embodiments, the display system 60 is the system 250of FIG. 6 , with FIG. 6 schematically showing some parts of that system60 in greater detail. For example, the waveguide assembly 260 of FIG. 6may be part of the display 70.

With continued reference to FIG. 9D, the display system 60 includes adisplay 70, and various mechanical and electronic modules and systems tosupport the functioning of that display 70. The display 70 may becoupled to a frame 80, which is wearable by a display system user orviewer 90 and which is configured to position the display 70 in front ofthe eyes of the user 90. The display 70 may be considered eyewear insome embodiments. In some embodiments, a speaker 100 is coupled to theframe 80 and configured to be positioned adjacent the ear canal of theuser 90 (in some embodiments, another speaker, not shown, may optionallybe positioned adjacent the other ear canal of the user to providestereo/shapeable sound control). The display system 60 may also includeone or more microphones 110 or other devices to detect sound. In someembodiments, the microphone is configured to allow the user to provideinputs or commands to the system 60 (e.g., the selection of voice menucommands, natural language questions, etc.), and/or may allow audiocommunication with other persons (e.g., with other users of similardisplay systems. The microphone may further be configured as aperipheral sensor to collect audio data (e.g., sounds from the userand/or environment). In some embodiments, the display system may alsoinclude a peripheral sensor 120 a, which may be separate from the frame80 and attached to the body of the user 90 (e.g., on the head, torso, anextremity, etc. of the user 90). The peripheral sensor 120 a may beconfigured to acquire data characterizing a physiological state of theuser 90 in some embodiments. For example, the sensor 120 a may be anelectrode.

With continued reference to FIG. 9D, the display 70 is operativelycoupled by communications link 130, such as by a wired lead or wirelessconnectivity, to a local data processing module 140 which may be mountedin a variety of configurations, such as fixedly attached to the frame80, fixedly attached to a helmet or hat worn by the user, embedded inheadphones, or otherwise removably attached to the user 90 (e.g., in abackpack-style configuration, in a belt-coupling style configuration).Similarly, the sensor 120 a may be operatively coupled by communicationslink 120 b, e.g., a wired lead or wireless connectivity, to the localprocessor and data module 140. The local processing and data module 140may comprise a hardware processor, as well as digital memory, such asnon-volatile memory (e.g., flash memory or hard disk drives), both ofwhich may be utilized to assist in the processing, caching, and storageof data. Optionally, the local processor and data module 140 may includeone or more central processing units (CPUs), graphics processing units(GPUs), dedicated processing hardware, and so on. The data may includedata a) captured from sensors (which may be, e.g., operatively coupledto the frame 80 or otherwise attached to the user 90), such as imagecapture devices (such as cameras), microphones, inertial measurementunits, accelerometers, compasses, GPS units, radio devices, gyros,and/or other sensors disclosed herein; and/or b) acquired and/orprocessed using remote processing module 150 and/or remote datarepository 160 (including data relating to virtual content), possiblyfor passage to the display 70 after such processing or retrieval. Thelocal processing and data module 140 may be operatively coupled bycommunication links 170, 180, such as via a wired or wirelesscommunication links, to the remote processing module 150 and remote datarepository 160 such that these remote modules 150, 160 are operativelycoupled to each other and available as resources to the local processingand data module 140. In some embodiments, the local processing and datamodule 140 may include one or more of the image capture devices,microphones, inertial measurement units, accelerometers, compasses, GPSunits, radio devices, and/or gyros. In some other embodiments, one ormore of these sensors may be attached to the frame 80, or may bestandalone structures that communicate with the local processing anddata module 140 by wired or wireless communication pathways.

With continued reference to FIG. 9D, in some embodiments, the remoteprocessing module 150 may comprise one or more processors configured toanalyze and process data and/or image information, for instanceincluding one or more central processing units (CPUs), graphicsprocessing units (GPUs), dedicated processing hardware, and so on. Insome embodiments, the remote data repository 160 may comprise a digitaldata storage facility, which may be available through the internet orother networking configuration in a “cloud” resource configuration. Insome embodiments, the remote data repository 160 may include one or moreremote servers, which provide information, e.g., information forgenerating augmented reality content, to the local processing and datamodule 140 and/or the remote processing module 150. In some embodiments,all data is stored and all computations are performed in the localprocessing and data module, allowing fully autonomous use from a remotemodule. Optionally, an outside system (e.g., a system of one or moreprocessors, one or more computers) that includes CPUs, GPUs, and so on,may perform at least a portion of processing (e.g., generating imageinformation, processing data) and provide information to, and receiveinformation from, modules 140, 150, 160, for instance via wireless orwired connections.

C. Examples of User Inputs

FIGS. 10A and 10B illustrate examples of user inputs received throughcontroller buttons or input regions on a user input device. Inparticular, FIGS. 10A and 10B illustrates a controller 3900, which maybe a part of the wearable system disclosed herein and which may includea home button 3902, trigger 3904, bumper 3906, and touchpad 3908. Theuser input device or a totem can serve as controller(s) 3900 in variousembodiments of wearable systems.

Potential user inputs that can be received through controller 3900include, but are not limited to, pressing and releasing the home button3902; half and full (and other partial) pressing of the trigger 3904;releasing the trigger 3904; pressing and releasing the bumper 3906;touching, moving while touching, releasing a touch, increasing ordecreasing pressure on a touch, touching a specific portion such as anedge of the touchpad 3908, or making a gesture on the touchpad 3908(e.g., by drawing a shape with the thumb).

FIGS. 10A and 10B illustrate various examples of user inputs that may bereceived and recognized by the system. The user inputs may be receivedover one or more modes of user input (individually, or in combination,as illustrated). The user inputs may include inputs through controllerbuttons such as home button 3902, trigger 3904, bumper 3906, andtouchpad 3908; physical movement of controller 3900 or HMD 3910; eyegaze direction; head pose direction; gestures; voice inputs; etc.

As shown in FIG. 10A a short press and release of the home button 3902may indicate a home tap action, whereas a long press of the home button3902 may indicate a home press & hold action. Similarly, a short pressand release of the trigger 3904 or bumper 3906 may indicate a triggertap action or a bumper tap action, respectively; while a long press ofthe trigger 3904 or bumper 3906 may indicate a trigger press & holdaction or a bumper press & hold action, respectively.

As shown in FIG. 10B, a touch of the touchpad 3908 that moves over thetouchpad may indicate a touch drag action. A short touch and release ofthe touchpad 3908, where the touch doesn’t move substantially, mayindicate a light tap action. If such a short touch and release oftouchpad 3908 is done with more than some threshold level of force(which may be a predetermined threshold, a dynamically determinedthreshold, a learned threshold, or some combination thereof), the inputmay indicate a force tap input. A touch of the touchpad 3908 with morethan the threshold level of force may indicate a force press action,while a long touch with such force may indicate a force press and holdinput. A touch near the edge of the touchpad 3908 may indicate an edgepress action. In some embodiments, an edge press action may also involvean edge touch of more than a threshold level of pressure. FIG. 10B alsoshows that a touch on touchpad 3908 that moves in an arc may indicate atouch circle action.

FIG. 10C illustrates examples of user inputs received through physicalmovement of a controller or a head-mounted device (HMD). As shown inFIG. 10C, physical movement of controller 3900 and of a head mounteddisplay 3910 (HMD) may form user inputs into the system. The HMD 3910can comprise the head-worn components 70, 110 shown in FIG. 9D. In someembodiments, the controller 3900 provides three degree-of-freedom (3DOF) input, by recognizing rotation of controller 3900 in any direction.In other embodiments, the controller 3900 provides six degree-of-freedom(6 DOF) input, by also recognizing translation of the controller in anydirection. In still other embodiments, the controller 3900 may provideless than 6 DOF or less than 3 DOF input. Similarly, the head mounteddisplay 3910 may recognize and receive 3 DOF, 6 DOF, less than 6 DOF, orless than 3 DOF input.

FIG. 10D illustrates examples of how user inputs may have differentdurations. As shown in FIG. 10D, certain user inputs may have a shortduration (e.g., a duration of less than a fraction of a second, such as0.25 seconds) or may have a long duration (e.g., a duration of more thana fraction of a second, such as more than 0.25 seconds). In at leastsome embodiments, the duration of an input may itself be recognized andutilized by the system as an input. Short and long duration inputs canbe treated differently by the wearable system. For example, a shortduration input may represent selection of an object, whereas a longduration input may represent activation of the object (e.g., causingexecution of an app associated with the object).

FIGS. 11A and 11B illustrate various examples of user inputs that may bereceived and recognized by the system. The user inputs may be receivedover one or more modes of user input (individually, or in combination,as illustrated). The user inputs may include inputs through controllerbuttons such as home button 3902, trigger 3904, bumper 3906, andtouchpad 3908; physical movement of controller 3900 or HMD 3910; eyegaze direction; head pose direction; gestures; voice inputs; etc.

As shown in FIG. 11A a short press and release of the home button 3902may indicate a home tap action, whereas a long press of the home button3902 may indicate a home press & hold action. Similarly, a short pressand release of the trigger 3904 or bumper 3906 may indicate a triggertap action or a bumper tap action, respectively; while a long press ofthe trigger 3904 or bumper 3906 may indicate a trigger press & holdaction or a bumper press & hold action, respectively.

As shown in FIG. 11B, a touch of the touchpad 3908 that moves over thetouchpad may indicate a touch drag action. A short touch and release ofthe touchpad 3908, where the touch doesn’t move substantially, mayindicate a light tap action. If such a short touch and release oftouchpad 3908 is done with more than some threshold level of force(which may be a predetermined threshold, a dynamically determinedthreshold, a learned threshold, or some combination thereof), the inputmay indicate a force tap input. A touch of the touchpad 3908 with morethan the threshold level of force may indicate a force press action,while a long touch with such force may indicate a force press and holdinput. A touch near the edge of the touchpad 3908 may indicate an edgepress action. In some embodiments, an edge press action may also involvean edge touch of more than a threshold level of pressure. FIG. 11B alsoshows that a touch on touchpad 3908 that moves in an arc may indicate atouch circle action.

Other example user input can include an input device (e.g., a user’shand) or mechanism, which may have six degrees of freedom.Advantageously, using a user’s hand as input can improve a userexperience by allowing a user to make intuitive pointing or other inputgestures to provide information to the AR system. In the case of objectinteraction and control, the use of a user’s hand or other pointingdevice can help a user more intuitively accommodate both targetingcomponents of an object and moving the object based on the position ororientation of the user’s hand (or other pointing device). Thus,discrete button activations may not be needed. However, in someexamples, a combination of discrete button activations and pointing witha six degree of freedom input device may be used.

D. Single Controller Content Movement and Interaction

Some applications implemented by an augmented reality (AR) or virtualreality (VR) system may include interactable and/or movable virtualcontent that accepts user input, through for example, user head pose,body pose, eye gaze, controller input, the like or a combinationthereof.. For example, an application may have a virtual control menu inwhich a user can select, highlight, or otherwise interact with the menuor information associated with the menu. In another example, anapplication may have a web browser in which a user can input informationwithin a browser window or control information using interactivefeatures, such as a refresh, home, or control button. An application mayalso allow the interactable virtual content to move within the user’senvironment as the user moves around their environment or by activeinput of the user. However, it can be difficult and clumsy for a user toboth interact with content and move the content within their environmentusing the sometimes limited controls provided by a VR or AR system. Forexample, if a user wants to interact with the interactable content, theymay provide one set of inputs and if a user wants to move theinteractable content, they may provide a second set of inputs. Combiningthe two sets of inputs can be clumsy and uncomfortable for a user.Described herein are systems and methods to simplify movement of and/orinteraction with interactable virtual content so as to make it lessburdensome to interact with the interactable virtual content.

One way to facilitate user interaction and movement of content in theuser’s environment is to separate the actions of interaction (e.g.,within a user interface) and movement (e.g., of the user interface). Forexample, an AR system may place the content at a fixed location withinthe user’s environment and then accept input from the user to interactwith the placed content based on user gestures, a controller, user gaze,the like, or some combination thereof. However, placing an interactivespace (e.g., a prism that includes a bounded volume of interactivecontent) at a set location within the user’s environment is not alwaysideal in the context of virtual or augmented reality. For example, in avirtual or augmented reality experience, a user may move around their 3Denvironment. If the interactive space is pinned at a set location, thenthe user would have to return to the location of the interactive spacein order to input information.

Another way to facilitate user interaction and movement of content is toutilize separate controllers for interaction and movement. For example,an AR system could have a controller for each hand of the user. A firstcontroller in the user’s first hand could move the interactive space inthe user’s 3D environment and the second controller in the user’s secondhand could select content within the interactive space. However, suchuse of two controllers results in an awkward and uncomfortable userexperience, in part, because it requires ambidextrous motions that usersare often not accustomed to.

Disclosed herein are systems and methods for an interaction and movementmechanic that allows both content placement and interaction using asingle controller. The interaction and movement mechanic can allow fordifferent magnitudes of movement by a controller to affect differentaspects of the input to the system. For examples, small movements withinan interactive content object (of, for example, a six degree of freedompointing device) can be used to target or select content within theinteractive content object. Larger movements outside of the interactivecontent object can cause the interactive content object’s position tofollow (or update/ respond to) the controlling device. Advantageously,such a system can allow a user to both bring content with them as theymove about their 3D environment and selectively interact with thatcontent using the same controller.

In general, the systems and methods described herein allow the user tomove a cursor or pointer (e.g., responsive to movement of a handheldcontroller) within an interactive content object, such as within aprism, to interact with the interactive content object. Once the usermoves the cursor or pointer outside of the interactive content object(or outside some additional boundary area surrounding the interactivecontent object), the functionality of the controller movement is updatedto move the position of the interactive content object to correspondwith movement of the controller, rather than to attempt to interact withcontent of the interactive content object. Thus, functionality of thecontroller may be alternated by the user providing movements of thecontroller. Depending on the embodiment, the threshold movement of thecontroller may be defined by a position of the controller pose withreference to a plane of the interactive content object and/or an anglebetween the controller pose and the plane of the interactive contentobject.

FIG. 12A illustrates an example content control environment 1200according to one example of a follow and interaction process disclosedherein. For example, a user 1202 may use a wearable device 1204 (such asa head mounted display disclosed above with reference to FIG. 9D) thatcan allow the user to perceive virtual content in their 3D environment.The wearable device 1204 may display virtual content 1206 at a locationin front of the user. The location in front of the user may bedetermined by a position and/or orientation of a controller 1208 of thewearable device (such as a user’s hand or the controller described withreference to FIGS. 10A-11B). As described in further detail below, ifthe user moves the controller 1208 so that a pointing vector of thecontroller does not intersect a bounded area or volume 1212 associatedwith the content 1206, then the wearable device 1204 may update thelocation of the content 1206 towards the new point of focus of thecontroller. If the user moves the controller 1208 so that the pointingvector of the controller intersects the bounded area or volume 1212,then the wearable device 1204 may allow the user to select and inputinformation associated with the content 1206 without moving the content1206. Advantageously, as the content 1206 moves in conjunction with thehead pose, an orientation of the content 1206 may be adjusted to match acurrent head pose of the user 1202. Thus, no matter where the content1206 is placed in the 3D environment of the user, the content is angledfor convenient viewing at the user’s current head pose. For example, insome implementations the content 1206 may be oriented perpendicular tothe head pose direction 1210.

Advantageously, as the user moves around their environment, the followand interaction processes disclosed herein can allow content to move andreorient with the user from location to location. For example, asillustrated in FIG. 12A, a user may have a first orientation andlocation 1201 a. The user may move to a new location 1201 b or 1201 cand have a new orientation. By tracking the user’s head pose andpointing vector associated with the controller 1208 as the user movesand reorients themselves, the AR system may move the location andorientation of the content 1206 and effectively follow the user to thenew location 1201 b or 1201 c. However, where the user makes smalleradjustments to the position and orientation of the controller 1206, theAR system may maintain the location of the content 1206 and allow theuser to provide input or interact with the content via those smallermovements of the controller 1206.

FIG. 12B shows a flowchart of an example process 1303 that may beutilized to control whether movements of the controller are used toupdate location of the content in the user’s 3D space or to allow theuser to interact with the content. A process 1303 can include somecombination of a direction determination block 1350, a content boundarydetermination block 1352, boundary comparison block 1354, a contentmovement block 1356, and an input identification block 1358, or fewer ormore blocks.

At a direction determination block 1350, the AR system may determine apointing vector of a controller, device, gesture, or other inputmechanism capable of indicating a direction. A pointing vector caninclude a direction within the 3D environment of the user that isindicated by one or more input devices, whether the input device iselectromechanical (e.g., a totem or handheld controller), mechanical, oran object (e.g., a user’s hand, finger, or pencil). In some examples,the pointing vector can include an indicated direction by one or moreuser gestures. In some examples, the pointing vector can include anindicated direction of a handheld controller, such as described abovewith reference to FIGS. 10A-11B. In some examples, a pointing vector maybe determined by more than one input, such as a controller orientationand user eye gaze or other combination of inputs.

At a content boundary determination block 1352, the AR system maydetermine a boundary of virtual content. The boundary may be one or moreedges of a volume of space associated with virtual content, some subsetof the volume of space (e.g., if the virtual content is very large), oran area that includes some space around a border of the virtual content.In some examples, the content boundary may be associated with more thanone piece of virtual content.

The content boundary may be of any shape. For example, the contentboundary may be a rectangular prism, sphere, truncated cone, or othershape. In some embodiments, the content boundary may or may not have ashape similar to that of the virtual content. For example, if thecontent is a rectangular menu, the content boundary can be rectangular.If the content is circular, the content boundary can be circular orrectangular. In some examples, the content boundary may be of the sameor similar bounds of the virtual content. For example, the virtualcontent may be a prism around a rectangular interactive menu. The prismmay have a rectangular prism having a length and height equal to orgreater than the interactive menu. In some embodiments, a contentboundary may be the edges of the rectangular prism.

Additionally or alternatively, the content boundary may be the same in ahorizontal and vertical direction or may be different. For example, thecontent boundary may be further away from the edges of the virtualcontent in the vertical direction than in the horizontal direction.Thus, the content may move more with less change in direction by thecontroller in the vertical direction than in the horizontal direction.

The content boundary may be smaller or larger than the bounds of thevirtual content or prism containing the virtual content. For example,the content boundary may coincide with the bounds of an interactiveportion of the virtual content (e.g., a menu tile) that may be smallerthan the full size of the virtual content. In some examples, the size orshape of the content boundary may vary based on one or more aspects ofthe content and/or AR system. For example, the content boundary may bedifferent for different types of virtual content or applicationsassociated with the virtual content. In some embodiments, the contentboundary can extend to include a portion or percentage of the user’sField of View. In some examples, the content boundary may be arectangular cuboid having a ⅓ meter, ½ meter, a meter, or other value.In other examples, the boundary may be 10%, 25%, or other amount of theuser’s field of view. In some examples, the virtual content may be sizedor adjusted to fit within the set threshold or boundary.

FIG. 12C illustrates example content 1206 within a bounded volume 1212and FIG. 12D illustrates a top view of a controller 1208 oriented suchthat its pointing vector is generally towards the content 1206 within abounded volume 1212. In this example, the bounded volume 1212 is largerthan the area 1230 of content 1206. Advantageously, this extra areaallows for a padding in which a user may change the direction of thepointing vector of the controller 1208 to outside the immediate area1230 of the content 1206 without triggering content movement (e.g.,without causing the content to follow movement of the controller). Thisextra area can be defined by an angular distance 1232 from the edges ofthe content area 1230. The angular distance 1232 can be any number ofdegrees, such as 5 degrees, 10 degrees, or 20 degrees. The angulardistance 1232 may be different at different edges of the area 1230 ofthe content 1206. For example, the angular distance 1232 can be greaterat a left or right edge than at a top or bottom edge of the content area1230. In another example, the angular distance 1232 can be differentbased on the content 1206. For example, the content 1206 may be a menuhaving more interactive content on the left side over the right side.Since a user interacting with content is more likely to move thepointing vector towards the side with more interactive content, the ARsystem may provide a greater padding on the left side to help avoidaccidental movement of the content 1206 to a new location and make iteasier for the user to interact with the content 1206.

With continued reference to FIG. 12B, at a boundary comparison block1354, the AR system may determine whether the pointing vector of thecontroller intersects the content boundary. For example, the pointingvector of the controller may pass within the content boundary or outsidethe content boundary. If the pointing vector intersects the contentboundary, then the AR system may move to block 1358. If the pointingvector does not intersect the content boundary, then the AR system maymove to block 1356.

At a content movement block 1356, the AR system may move the virtualcontent to a new location. For example, the AR system may move thecontent towards a pointing vector of the controller. The point may bedetermined based on one or more factors associated with the AR system orapplication, such as described in further detail below. The AR systemmay move the content so that the pointing vector just beginsintersecting with the content boundary or may move the content so thatthe pointing vector intersects a particular point within the content,such as the center of the content.

The AR system may move the content at a constant or variable speed. Forexample, the AR system may move the content at a speed calculated basedon the distance between the current content location and the desiredcontent location. In some examples, the speed may be faster for afurther distance and slower for a smaller distance. In some examples,the speed may be variable. For example, the content may move slowly atfirst, speeds up, and then slows down closer to the destination.

At an input identification block 1358, the AR system may receive userinput associated with the content. For example, the user may indicatewith the controller (and/or gesture, voice command, or the like)interactions with interactive content (rather than continued movement ofthe interactive content in relation to controller movements). The ARsystem may receive the indication and perform actions based on theindication. For example, the content can include a virtual menu havingselectable buttons. The AR system may receive an indication from theuser to select one or more of the selectable buttons and perform one ormore actions based on the selection.

E. Example Content Follow Movement

As mentioned above, an AR system may control the movement of virtualcontent within a 3D environment through the manipulation of acontroller, resulting in the content effectively following the user asthe user moves around their space. FIGS. 13A, 13B, and 13C illustrateexample aspects of example content movement environments that mayinclude moving virtual content at a content location 1304 with acontroller 1302.

A controller 1302 can be any multiple degree of freedom input device.For example the controller 1302 can include a user’s hand or portionthereof, a controller of a wearable device, such as described above withreference to FIGS. 10A-11B, a six degree of freedom device capable ofproviding a pointing direction, a three degree of freedom touch inputdevice, the like, or some combination thereof. A controller 1302 mayhave some combination of sensors, controls, or output components thatallow the controller 1302 to indicate a direction. In some examples, theindicated direction may coincide with a direction parallel to an axis ofthe controller 1302. In some examples, a controller 1302 may indicate adirection using an orientation or positioning of the controller 1302.The indicated direction may then define the direction of a pointingvector of the controller 1302.

Virtual content can include one or more prisms, which generallydescribes a three-dimensional container, area, or volume associated withmixed reality content, which may contain multiple virtual content items,such as representations of 3D objects. As illustrated in FIG. 13B, aprism can include a bounded volume 1318 that may have a width w and aheight h and another dimension. Content bounded in the prism may becontrolled or placed in the user’s environment by controlling or placingthe prism in which the content is bounded. A virtual object, as usedherein, may be or include a prism. Various characteristics, uses, andimplementations of prisms are described in U.S. Pat. Publication No.2019/0197785, published Jun. 27, 2019, which is hereby incorporated byreference herein in its entirety. In one example, content within theprism may include a menu associated with an application. For example,the virtual content can include a control menu capable of accepting userinput through input at one or more locations within the menu.

The virtual content may be centered at a location 1304 within the 3Denvironment of the user. The location 1304 may be based on one or morefactors associated with the user, the wearable device, or the user’senvironment. For example, the location 1304 can be relative to thelocation of the controller 1302. The controller 1302 may define acoordinate system with a point of origin at a point on the controller1302. In cases where the location 1304 is relative to a controller 1302or other point of reference, the location 1304 can fall within a rangeof distances to the point of origin associated with the controller 1302or other point of reference along the pointing vector 1312. For example,a vector 1322, such as illustrated in FIG. 13B, may be defined from thepoint of origin of the controller 1302 to a content location (orreference point associated with the content), and the location 1304 maybe along the vector 1322.

With reference to FIG. 13A, in determining a content location 1304, anAR system may calculate or determine a location 1306 of the contentalong a pointing vector 1312 of the controller 1302. For example, the ARsystem may define a coordinate system having a point of origin at thecontroller 1302. The coordinate system may have a z axis parallel topointing vector 1312 of the controller 1302. The content location 1304can be defined within a coordinate system of the controller 1302. Assuch, the content location 1304 may have a z component (such as z1 asillustrated in FIG. 13A). The z component may correspond to the contentlocation 1306 along the pointing vector 1312 of the controller 1302. Ifthe z component falls below a minimum distance 1307, then the AR systemmay move the content from location 1304 along pointing vector 1312 untilthe z component is greater than or equal to the minimum distance. If thez component falls above a maximum distance 1308, then the AR system maymove the content from location 1304 along the pointing vector 1312 untilthe z component is less than or equal to the maximum distance.

A content location 1304 may additionally have a point of reference withrespect to the width w and height h of bounded volume 1318. A pointingvector 1312 of the controller 1302 may intersect with the bounded volume1318 having a height h and width w at a point of intersection 1314. If alocation of point 1314 falls outside the width, w, of the bounded volume1318, then the AR system may move the content from location 1304horizontally until the distance is within bounds. If the verticaldistance exceeds the bounding height, h, of the bounded volume 1318, theAR system may move the content from location 1304 vertically until thedistance is within bounds. As such, if the controller 1302 rotates ormoves horizontally or vertically within the bounded volume 1318, thelocation 1304 of the bounded volume 1318 may stay relatively fixed inspace. However, if the controller 1302 moves horizontally or verticallyoutside the bounded volume 1318, the AR system may move the virtualcontent and/or bounded volume 1318 to a new location in space 1314.

FIG. 13C illustrates another example content movement environment 1301.In this example content movement environment 1301, an AR system maymaintain the content 1362 between a minimum and maximum distance awayfrom a controller 1302. For example, if the distance between the content1362 and the controller 1302 is less than the minimum, then the ARsystem may set the distance to the minimum. Additionally oralternatively, if the distance between the content 1262 and thecontroller 1302 is greater than the maximum, then the AR system may setthe distance to the maximum distance. Otherwise, the AR system maymaintain the current distance between the content 1362 and thecontroller 1302, making no adjustments.

Additionally or alternatively, a user’s ability to easily locate andinteract with the content 1362 may be improved if the content 1362remains in front of the controller 1302, e.g., so the controller pointsto a spot inside the bounds of the content 1362. To help achieve themaintained distance and allow the user to point to a spot inside thecontent 1362 or prism, the AR system may move the content (e.g., a prismor other three-dimensional content item) on the surface of an invisiblesphere 1364.

With reference to FIG. 13C, an invisible sphere 1364 may have a center1360 at a point 1360 on the controller 1302 (or some other point on thecontroller). The AR system may convert the location of the content 1362into spherical coordinates having an origin at the point 1360. Thespherical coordinates may include a distance, an azimuthal angle ϕ(e.g., rotation in the horizontal plane) and a polar angle ⊖ (e.g.,similar to pitch). In other embodiments, another coordinate system maybe used.

As with the discussion above, the content 1362 (e.g., a prism or otherthree-dimensional content) may have a set of bounds. The set of boundsmay also be associated with spherical coordinates with reference to thedesignated center 1360. In this example, the AR system may determine theouter surface of the sphere 1364 with reference to the bounds of thecontent 1362 (e.g., so the bounds of the content are maintained adjacenta surface of the sphere 1364). In one particular example, the sphericalcoordinates are determined, with C representing the controller locationand P representing the content (e.g., prism) location so that: a vector,CP, from the controller 1302 to the content 1362 is defined as:

CP = P − C

where the distance between the controller 1302 and the content 1362 isthe length of CP, an azimuth of the vector is the arctangent of CP.z /CP.x, a horizontal distance of CP is the distance of the vectorprojected onto the X-Z plane, which may be calculated as hypot(CP,x,CP.y), and an altitude of the content location is the arctangent of theheight / horizontal distance or arctan(CP.y /horizontal distance). Insome embodiments, the location of the content 1362 may be an anchorassociated with the prism, such as a point at the center of the prism’svolume or a point at the center (or other location) of one of the sidesof the prism, for example that may indicate a location where the prismand hence the content can be attached to other real or virtual objects,such as walls, table, chair, or the like.

Advantageously, in some embodiments, as the controller is moved, theposition of the content 1362 is adjusted to remain on the outer surfaceof the sphere 1364. For example, if a current azimuth and polar angle ofthe controller pointing vector falls inside the bounded area of thesphere, then the AR system may maintain the position of the content1362. If a current azimuth and/or polar angle of the pointing vectorfalls outside the bounded area of the sphere, then the AR system mayupdate the position of the content 1362. For example, the AR system maydetermine a difference in azimuth between the nearest point on thebounded area and the current azimuth of CP. The AR system may then movethe content 1362 to reduce the difference. Additionally oralternatively, the AR system may adjust the altitude of the content toreduce a difference in altitude of the content location and CP.

In one particular example, a position of content 1360 may be adjustedfor distance (e.g., as noted above with reference to a minimum andmaximum distance), and with reference to azimuth and altitude. Todetermine if the azimuth angle of the content needs to be adjusted, thehorizontal bounds may be converted to an angle, e.g., the arctangent ofhalf the horizontal bounds divided by the distance: arctan((horiz_bounds / 2) / distance ). If the difference in azimuth is greaterthan this angle, then the AR system may reduce the difference in azimuthto this angle. To determine if the altitude angle of the content needsto be adjusted, the vertical bounds may be converted to an angle, e.g.the arctangent of half the vertical bounds divided by the distance:arctan( (vert_bounds / 2) / distance). If the difference in altitude isgreater than this angle, then the AR system may reduce the difference inaltitude to this angle.

In some examples, the AR system may transfer the coordinates of thecontent 1360 into Cartesian coordinates. Previously, the sphericalcoordinates of the content were in reference to the controller.Advantageously, using Cartesian coordinates can allow the AR system toposition the content within the environment of the user withoutreference to the controller 1302. To convert to Cartesian (rectangular)coordinates from spherical coordinates, the AR system may apply thefollowing formulas:

-   X = distance * cos(azimuth) * cos(altitude)-   Y = distance * sin(altitude)-   Z = distance * sin(azimuth) * cos(altitude)

Where X corresponds to an x coordinate, Y corresponds to a y coordinatein a Cartesian coordinate frame, and Z corresponds to a z coordinate ina Cartesian coordinate frame. F. Example Content Orientation

An AR system may be capable of controlling the orientation of virtualcontent within the 3D environment of the user. FIGS. 14A-14C illustrateaspects of an example content orientation environment 1400 forcontrolling the orientation of virtual content 1404 that includes acontroller 1302 and head mounted device 1402.

With reference to FIG. 14A, a content orientation environment 1400 caninclude a user 1406 wearing a head mounted display 1402 and/or using acontroller 1302 to manipulate and/or view virtual content 1404. The usermay view the virtual content 1404 at a location 1412 in the 3Denvironment of the user 1406. The content may be displayed at an angle1410 with respect to a gaze vector 1405 associated with the user’s headpose. In some examples, the angle 1410 may be optimized so that the usercan more easily perceive a surface of the content 1404. In someexamples, the angle 1410 may be such that a surface of the content 1404is perpendicular to the gaze direction of the user.

FIG. 14B illustrates a top-down view of an example content orientationenvironment 1401. As shown in FIG. 14B, the content 1420 may be rotatedso that it faces (e.g., remains perpendicular to a controller pose of)the controller 1302. Thus, the surface of the content 1420 facing thehead mounted display 1402 may be rotated to an angle 1422 with respectto an x axis of a coordinate system centered at the head mounted displayof the user. The angle 1422 may be updated as the controller moves sothat the surface 1420 of the content remains locked in a perpendicularangle to a pointing vector of the controller 1302. In another example,the angle 1422 may be updated upon user or application input.

FIG. 14C illustrates a side view of the example content orientationenvironment 1401. As shown in FIG. 14C, the content 1420 may be rotatedso that it faces the head of the user or head-mounted display 1402.Thus, the surface of the content 1420 facing the head mounted display1402 or head of the user may be rotated to an angle 1432 with respect toa y axis of a coordinate system centered at the head mounted display ofthe user. The angle 1432 may be updated as the user moves their head,eye gaze, and/or head mounted display so that the surface of the content1420 remains locked in a perpendicular angle to a gaze vector 1405associated with the user’s head pose. In another example, the angle 1432may be updated upon user or application input. In some examples, theangle 1432 may be based on the height of the content 1420 such thatsurface of the content 1420 may be oriented to face an origin pointassociated with the user, such as a point on the user’s head. In anotherexample, the angle 1432 and/or height of the content 1420 may be fixed.

FIG. 14D illustrates a flow chart of an example content orientationprocess 1403. For example, a content orientation process 1403 caninclude a content location determination block 1442, a head posedetermination block 1444, a content orientation block, or more or fewerblocks.

At a content location determination block 1442, the AR system maydetermine the location of virtual content within the 3D environment ofthe user. For example, the virtual content may be located at a point infront of the user (such as at a location determined by movement rulesdescribed with reference to FIGS. 12A-13B). The AR system may determinethe content location based on input from or data associated with thevirtual content, application, and/or AR system.

At a head pose determination block 1444, the AR system may determine thehead pose of the user. For example, the AR system may detect one or moreparameters associated with the head pose of the user using one or moresensors associated with the AR system, such as one or moreoutward-facing cameras associated with a head-mounted display worn bythe user, an inertial measurement unit, some combination thereof orother sensors. The head pose of the user may be utilized to helpdetermine a gaze direction of the user.

At a content orientation block 1446, the AR system may utilize thecontent location and gaze direction of the user to reorient the virtualcontent. For example, the AR system may orient a surface of the contentwhile at the determined content location to be perpendicular to the gazedirection of the user. In some examples, the AR system may additionallymove the location of the content so as to accomplish a comfortableviewing experience of the user in viewing the content at the updatedorientation.

G. Example Content Control Mechanics

FIG. 15 illustrates an example content access mechanic and FIG. 16illustrates an example content drop mechanic that may be utilized inconjunction with the interaction and movement mechanic described above.

1. Menu Open and Close

In some examples, content may include a control menu. FIG. 15illustrates an example menu access process 1500 for accessing a controlmenu. With reference to FIG. 15 , a menu access process 1500 can includean indication block 1502, a direction determination block 1504, an openlocation determination block 1506, an animation block 1508, a menudisplay block 1510, fewer, or more blocks. While the example of FIG. 15is discussed with reference to a control menu, the process may be usedwith other content, such as a prism that includes one or moreinteractive virtual objects.

At an indication block 1502, the AR system may receive an indication toopen or access a menu or other content. The indication can include aninput, gesture, or pose. For example, the input can include pointing, apress of a button or other input component of a controller associatedwith the AR system. In another example, the input can include a gestureassociated with accessing the menu or other content. In the case of aninput that does not involve the direct press of a button or other directinput to a controller, the AR system may perceive the input using one ormore sensors associated with the AR system, such as an outward facingimaging system of a head-mounted display. For example, an input caninclude six degree of freedom pointing by a user’s hand or pointingdevice. In another example, the input can include input to a multipledegree of freedom touchpad. The AR system may then determine whether anindication is given based on gestures, poses, or other indirect ordirect input detected by the one or more sensors. In some examples,different inputs may bring up different types of menus or differentcontent. In some examples, different inputs may be used to bring up thesame menu or content.

At a direction determination block 1504, the AR system may determine apointing vector of a controller, device, gesture, or other inputmechanism capable of indicating a direction. A pointing vector caninclude a direction within the 3D environment of the user that isindicated by one or more input devices. In some examples, the pointingvector can include an indicated direction by one or more user gestureswith a six degree of freedom input device or the user’s hand. In anotherexample, the pointing vector can include an indicated direction by athree degree of freedom touch input. In some examples, the pointingvector can include an indicated direction of a handheld controller, suchas described above with reference to FIGS. 10A-11B. In some examples, apointing vector may be determined by more than one input, such as acontroller orientation and user eye gaze or other combination of inputs.

At an open location determination block 1506, the AR system maydetermine or identify a location to open the indicated menu or otherwisedisplay content within the 3D environment of the user. The location maybe based on the determined direction of focus or other input from theuser, application, or AR system. For example, the user may point thecontroller in their environment. The AR system may display the menu orother virtual content at a point along the pointing vector. The pointalong the pointing vector may be determined based on any number offactors, including but not limited to one or more rules associated withcontent location, such as discussed with reference to FIGS. 12A-13B. Insome examples, the open location may be at a set distance away from thecontroller along the pointing vector. In another example, the openlocation may be in the center of the user’s field of view.

At an animation block 1508, the AR system may generate animations,sounds, feedback, or other effects to indicate the opening location ofthe menu or other content to the user. For example, the open locationmay be at a point along the direction of focus ray. The direction offocus ray may be pointed towards an area of the 3D environment of theuser that is not currently being perceived by the user with their headmounted display. In order to draw the attention of the user to theopening location of the menu or other content, the AR system may displayan animation, generate haptic feedback, or play sounds indicating thecontent is going to open at the location. For example, the AR system maydisplay sparks or other content coming out of the controller or otherreference point along the direction of focus towards the open location.Thus, the user’s gaze or attention may be more likely drawn to the menuor content location.

At a menu display block 1510, the AR system may display the menu at theopen location at an orientation comfortable for the user to view themenu or other interactive virtual content. For example, the orientationand/or position of the menu or other content may be displayed and/orupdated according to one or more processes described above withreference to FIGS. 12A-14D.

Additionally or alternatively, the AR system may close or cease todisplay the content or menu based on one or more closing indications. Aclosing indication can include an input to a controller, gesture,command, other input, or some combination of inputs to cease to displaythe menu. Upon receipt of a closing indication, the AR system may ceaseto display the menu or other content.

2. Content Dropping

FIG. 16 illustrates an example menu access process 1600 for stopping andstarting a content follow movement, such as described above withreference to FIGS. 12A and 12B. With reference to FIG. 16 , a contentdropping process 1600 can include a location determination block 1602, afollow decision block 1604, a display block 1606, a follow decisionblock 1608, and a follow block 1610.

At a location determination block 1602, the AR system can determine acurrent location of virtual content or menu being manipulated. Forexample, the virtual content or menu may be moved to a designatedlocation associated with a direction of focus, such as described abovewith reference to FIGS. 12A-13B. The AR system may determine thatupdated location or other location associated with the user or theuser’s 3D environment to identify where content is or should be located.In some examples, the location may be updated as the user walks around aroom as the location of the controller and orientation of the directionof focus changes. In some embodiments, the content may be placed at afixed distance in front of the user.

At a follow decision block 1604, the AR system can determine whether theAR system should continue to update the location of the content. Forexample, the AR system may identify a stopping condition for continuingto update the location of the content (or follow process or mechanic). Astopping condition can include an indication from the user or othersource to stop updating the location of the content. For example, a usercan gesture, issue a command, or press a button or other input on acontroller to stop the content from following the user. In anotherexample, the AR system may identify that the user has exited a boundedvolume or exceeded a threshold follow condition. In some examples, acombination of stopping conditions may be used. If a stopping conditionis detected, the AR system may move to block 1606. If a stoppingcondition is not detected, the AR system may move to block 1610 tocontinue updating the location of the content (in other words, continuethe follow process or mechanic), such as described above with referenceto FIGS. 12A-14B.

At a display block 1606, the AR system can display the content at acurrent or designated location of the 3D environment of the user. Forexample, if the AR system receives an indication to stop following atblock 1604, the AR system may drop the menu or other content at thecurrent location of the menu or content such that the menu or contentstays at the last updated location. In some examples, the AR system maycontinue to update the orientation of the menu or content, such asdescribed above with reference to FIGS. 14A-14B. In some examples, theAR system may also freeze the orientation of the menu or content to acurrent orientation. In some examples, the AR system may freeze theorientation of the menu or content as a result of detecting a freezecondition, such as a gesture, command, press a button, or otherwiseprovide input to the AR system to indicate to the AR system to freeze orstop updating an orientation of the content. In some examples, when theAR system receives a stopping condition at a block 1610, the AR systemmay update the orientation to a fixed or preset orientation and/orheight within the 3D environment of the user. For example, the AR systemmay fix the content at eye height and fix the orientation of the contentfor a surface of the content to be perpendicular to the floor of the 3Denvironment of the user such that the user can view the content withease.

At a follow decision block 1608, the AR system can determine whether theAR system should continue to update the location of the content. Forexample, the AR system may identify a starting condition for updatingthe location of the content (or follow process or mechanic). A startingcondition can include an indication from the user or other source tostart updating the location of the content. For example, a user cangesture, issue a command, press a button or other input to start havingthe content follow the user. In another example, the AR system mayidentify that the user has entered a bounded volume or passed athreshold follow condition. In some examples, a combination of startingconditions may be used. If a starting condition is detected, the ARsystem may move to block 1610 to continue updating the location of thecontent (in other words, continue the follow process or mechanic), suchas described above with reference to FIGS. 12A-14B.

In some examples, when the AR system receives an indication to beginupdating the location and/or orientation of the content, the AR systemmay summon the content to the current location of the user. For example,the AR system may move the current location of the content to a newlocation closer to the user. In some examples, the new location may bebased on the one or more rules regarding location of the contentdescribed with reference to FIGS. 12A-14B above. In some examples, theAR system may not update the location and/or orientation of the contentto summon the content to the current location of the user until a summonindication is detected. A summon indication can include a gesture, acommand, a button press, or other input.

H. Example Application of Content Follow System

FIG. 17 depicts an example application 1700 for a content follow systemwhere two users of respective wearable systems are conducting atelepresence session. Two users (named Alice 912 and Bob 914 in thisexample) are shown in this figure. The two users are wearing theirrespective wearable devices 902 and 904 which can include an HMDdescribed with reference to FIG. 9D (e.g., the display device 70 of thesystem 60) for representing a virtual avatar of the other user in thetelepresence session. The two users can conduct a telepresence sessionusing the wearable device. Note that the vertical line in FIG. 17separating the two users is intended to illustrate that Alice and Bobmay (but need not) be in two different locations while they communicatevia telepresence (e.g., Alice may be inside her office in Atlanta whileBob is outdoors in Boston).

The wearable devices 902 and 904 may be in communication with each otheror with other user devices and computer systems. For example, Alice’swearable device 902 may be in communication with Bob’s wearable device904, e.g., via the network 990. The wearable devices 902 and 904 cantrack the users’ environments and movements in the environments (e.g.,via the respective outward-facing imaging system 464, or one or morelocation sensors) and speech (e.g., via the respective audio sensor232). The wearable devices 902 and 904 can also track the users’ eyemovements or gaze based on data acquired by the inward-facing imagingsystem 462. In some situations, the wearable device can also capture ortrack a user’s facial expressions or other body movements (e.g., arm orleg movements) where a user is near a reflective surface and theoutward-facing imaging system 464 can obtain reflected images of theuser to observe the user’s facial expressions or other body movements.

A wearable device can use information acquired of a first user and theenvironment to animate a virtual avatar that will be rendered by asecond user’s wearable device to create a tangible sense of presence ofthe first user in the second user’s environment. For example, thewearable devices 902 and 904, the remote computing system 920, alone orin combination, may process Alice’s images or movements for presentationby Bob’s wearable device 904 or may process Bob’s images or movementsfor presentation by Alice’s wearable device 902. As further describedherein, the avatars can be rendered based on contextual information suchas, e.g., a user’s intent, an environment of the user or an environmentin which the avatar is rendered, or other biological features of ahuman.

Although the examples only refer to two users, the techniques describedherein should not be limited to two users. Multiple users (e.g., two,three, four, five, six, or more) using wearables (or other telepresencedevices) may participate in a telepresence session. A particular user’swearable device can present to that particular user the avatars of theother users during the telepresence session. Further, while the examplesin this figure show users as standing in an environment, the users arenot required to stand. Any of the users may stand, sit, kneel, lie down,walk or run, or be in any position or movement during a telepresencesession. The user may also be in a physical environment other thandescribed in examples herein. The users may be in separate environmentsor may be in the same environment while conducting the telepresencesession. Not all users are required to wear their respective HMDs in thetelepresence session. For example, Alice may use other image acquisitionand display devices such as a webcam and computer screen while Bob wearsthe wearable device 904.

Bob may provide an indication through, for example, a button press on acontroller 1704, to display a chat menu or other content 1702 during theavatar chat session. Bob’s AR system may display the content 1702according to, for example, the access process 1500 described above withreference to FIG. 15 . The content 1702 may appear to Bob at a locationalong a pointing vector 1706 in Bob’s environment. The content 1702 maybe oriented to allow Bob to view the content 1702 according to, forexample, the orientation process described with reference to FIG. 14Dabove. The content 1702 may or may not be visible to Alice. In exampleswhere the content 1702 is visible to Alice, the content 1702 may or maynot be oriented so that Alice can view the content 1702. As Bob movesaround his space during Avatar chat and reorients his controller or headmounted display within the 3D environment, the content 1702 may followBob according to one or more processes described above. In someexamples, Bob may place the chat menu or other content 1702 at a desiredlocation within Bob’s environment. Additionally or alternatively, Bobmay select or interact with portions of the content 1702 using smallermovements of the controller 1704. Bob may close the chat menu or othercontent 1702 through one or more indications to Bob’s AR system.

I. Additional Examples

Disclosed herein are additional examples of an AR system. Any of theexamples disclosed may be combined.

Example 1: An augmented reality (AR) system comprising:

-   an AR display configured to present virtual content to a user of the    AR system;-   an outward facing camera configured to capture one or more images of    an environment of the user;-   a handheld controller defining a pointing vector indicating a    pointing direction of the handheld controller;-   a hardware processor in communication with the AR display, the    outward facing camera, and the handheld controller, the hardware    processor programmed to:    -   display an interactive content object via the AR display;    -   in a first interaction mode, while the pointing vector remains        within the interactive content object, indicating movements of        the handheld controller within the interactive content object        and allowing interactions with the interactive content object        via the handheld controller;    -   monitoring changes of the pointing vector with reference to the        interactive content object;    -   in response to detecting movement of the pointing vector outside        the interactive content object, updating the system to a second        interaction mode wherein the handheld controller causes movement        of the interactive content object such that the interactive        content object follows movements of the handheld controller in        the virtual environment; and    -   in response to detecting movement of the pointing vector of the        handheld controller inside the interactive content object,        updating the system to the first interaction mode.

Example 2: The system of Example 1, wherein the interactive contentobject comprises a prism containing a virtual object.

Example 3: The system of any one of Examples 1 or 2, wherein at leastone edge of the interactive content object is 10 degrees further awayfrom the center of the virtual object than a corresponding edge of thevirtual object.

Example 4: The system of any one of Examples 1-3, wherein the hardwareprocessor is configured to receive an indication to display theinteractive content at a first content location.

Example 5: The system of Example 4, wherein the indication comprises apress and release of a button on the controller.

Example 6: The system of any one of Examples 4-5, wherein the hardwareprocessor is configured to alert the user to the display of interactivecontent at the first content location.

Example 7: The system of Example 6, wherein the alert comprises at leastone of: graphics, haptic feedback, or sounds.

Example 8: The system of any one of Examples 1-7, wherein the hardwareprocessor is configured to receive an indication to stop displaying theinteractive content in the environment of the user.

Example 9: The system of Example 8, wherein the indication comprises apress and release of a button on the controller.

Example 10: The system of any one of Examples 1-9, wherein the hardwareprocessor is configured to:

-   determine a first user head pose;-   identify a first user gaze vector based on the first user head pose;    and-   orient the interactive content so that a surface of the interactive    content object is perpendicular to the first user gaze vector.

Example 11: The system of Example 10, wherein the hardware processor isconfigured to:

-   determine a second user head pose;-   identify a second user gaze vector based on the second user head    pose; and-   orient the interactive content object so that a surface of the    interactive content object is perpendicular to the second user gaze    vector.

Example 12: The system of Example 1-11, wherein a pitch of theinteractive content object is fixed with respect to a head height of theuser.

Example 13: The system of any one of Examples 1-12, wherein the hardwareprocessor is configured to:

-   receive an indication to stop moving the interactive content; and-   in response to receiving the indication to stop moving, maintain    display of the interactive content at a current content location    within the environment of the user.

Example 14: The system of Example 13, wherein the hardware processor isconfigured to:

-   receive an indication to begin moving the interactive content; and-   in response to receiving the indication to begin moving, allow the    interactive content to be displayed at a new location within the    environment of the user.

Example 15: An augmented reality (AR) system comprising:

-   an AR display configured to present virtual content to a user of the    AR system;-   a handheld controller having at least six degrees of freedom; and-   a hardware processor in communication with the AR display, the    outward facing camera, and the handheld controller, the hardware    processor programmed to:    -   display interactive content at a first content location;    -   determine a first pointing vector comprising a direction        indicated by the controller;    -   determine whether the first pointing vector intersects with a        bounded volume associated with the interactive content;    -   in response to determining that the first pointing vector does        not intersect the bounded volume, move the interactive content        to a second content location associated with a point along the        direction of the first pointing vector; and    -   in response to determining that the first pointing vector        intersects the bounded volume, receive an indication to interact        with the interactive content at the first content location.

Example 16: The system of Example 15, wherein the interactive contentcomprises a prism containing a virtual object.

Example 17: The system of any one of Examples 15 or 16, wherein at leastone edge of the interactive content is 10 degrees further away from thecenter of the virtual object than a corresponding edge of the virtualobject.

Example 18: The system of Example 15, wherein the hardware processor isconfigured to receive an indication to display the interactive contentat the first content location.

Example 19: The system of Example 18, wherein the indication comprises apress and release of a button on the controller.

Example 20: The system of any one of Examples 18-19, wherein thehardware processor is configured to alert the user to the display ofinteractive content at the first content location.

Example 21: The system of Example 20, wherein the alert comprises atleast one of: graphics, haptic feedback, or sounds.

Example 22: The system of any one of Examples 15-21, wherein thehardware processor is configured to receive an indication to stopdisplaying the interactive content in the environment of the user.

Example 23: The system of Example 22, wherein the indication comprises apress and release of a button on the controller.

Example 24: The system of any one of Examples 15-24, wherein thehardware processor is configured to:

-   determine a first user head pose;-   identify a first user gaze vector based on the first user head pose;    and-   orient the interactive content so that a surface of the interactive    content is perpendicular to the first user gaze vector.

Example 25: The system of Example 24, wherein the hardware processor isconfigured to:

-   determine a second user head pose;-   identify a second user gaze vector based on the second user head    pose; and-   orient the interactive content so that a surface of the interactive    content is perpendicular to the second user gaze vector.

Example 26: The system of any one of Examples 15-25, wherein the firstcontent location and the second content location are at the same heightfrom the ground.

Example 27: The system of Example 26, wherein a pitch of the interactivecontent is fixed with respect to a head height of the user.

Example 28: The system of any one of Examples 15-27, wherein thehardware processor is configured to:

-   receive an indication to stop moving the interactive content; and-   in response to receiving the indication to stop moving, maintain    display of the interactive content at a current content location    within the environment of the user.

Example 29: The system of Example 28, wherein the hardware processor isconfigured to:

-   receive an indication to begin moving the interactive content; and-   in response to receiving the indication to begin moving, allow the    interactive content to be displayed at a new location within the    environment of the user.

Example 30: A method for displaying virtual content, the methodcomprising:

-   displaying interactive content at a first content location;-   determining a first pointing vector comprising a direction indicated    by a controller;-   determining whether the first pointing vector intersects with a    bounded volume associated with the interactive content;-   in response to determining that the first pointing vector does not    intersect the bounded volume, moving the interactive content to a    second content location associated with a point along the direction    of the first pointing vector; and-   in response to determining that the first pointing vector intersects    the bounded volume, receiving an indication to interact with the    interactive content at the first content location.

Example 31: The method of Example 30 comprising receiving an indicationto display the interactive content at the first content location.

Example 32: The method of Example 31, wherein the indication comprises apress and release of a button on the controller.

Example 33: The method of any one of Examples 31-32 comprisingcommunicating an alert to the user associated with the display ofinteractive content at the first content location.

Example 34: The method of Example 33, wherein the alert comprises atleast one of: graphics, haptic feedback, or sounds.

Example 35: The method of any one of Examples 30-34 comprising receivingan indication to stop displaying the interactive content in theenvironment of the user.

Example 36: The method of Example 35, wherein the indication comprises apress and release of a button on the controller.

Example 37: The method of any one of Examples 30-36 comprising:

-   determining a first user head pose;-   identifying a first user gaze vector based on the first user head    pose; and-   orienting the interactive content so that a surface of the    interactive content is perpendicular to the first user gaze vector.

Example 38: The method of Example 37 comprising:

-   determining a second user head pose;-   identifying a second user gaze vector based on the second user head    pose; and-   orienting the interactive content so that a surface of the    interactive content is perpendicular to the second user gaze vector.

Example 39: The method of any one of Examples 30-38, wherein the firstcontent location and the second content location are at the same heightfrom the ground.

Example 40: The method of Example 39, wherein a pitch of the interactivecontent is fixed with respect to a head height of the user.

Example 41: The method of any one of Examples 30-40 comprising:

-   receiving an indication to stop moving the interactive content; and-   in response to receiving the indication to stop moving, maintaining    display of the interactive content at a current content location    within the environment of the user.

Example 42: The method of Example 41 comprising:

-   receiving an indication to begin moving the interactive content; and-   in response to receiving the indication to begin moving, allowing    the interactive content to be displayed at a new location within the    environment of the user.

Example 43: Any of the above examples wherein the interactive contentobject comprises a bounded volume containing a virtual object.

J. Other Considerations

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, animationsor video may include many frames, with each frame having millions ofpixels, and specifically programmed computer hardware is necessary toprocess the video data to provide a desired image processing task orapplication in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps or actions may be stored,persistently or otherwise, in any type of non-transitory, tangiblecomputer storage or may be communicated via a computer-readabletransmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

1. An augmented reality (AR) system, comprising: an AR displayconfigured to present virtual content to a user of the AR system; one ormore sensors configured to sense movement of the user; and a hardwareprocessor in communication with the AR display and the one or moresensors, the hardware processor programmed to: receive a firstindication via the one or more sensors; determine a first pointingvector that indicates a first direction within a three-dimensional (3D)environment of the user based on the first indication received via theone or more sensors; determine a first location within the 3Denvironment of the user based on the first pointing vector; and displayon the AR display a first interactive content object based on the firstlocation in the 3D environment of the user, wherein the firstinteractive content object is a control menu or a prism that includesone or more interactive virtual objects.
 2. The system of claim 1,wherein the first indication is a gesture or pose of the user.
 3. Thesystem of claim 1, wherein the one or more sensors include an outwardfacing camera, wherein the first indication is received via the outwardfacing camera, and wherein the first location is in a center of a fieldof view of the user.
 4. The system of claim 1, wherein the firstindication is a press of a button of a handheld controller, wherein thefirst direction within the 3D environment is an indicated direction ofthe handheld controller, and wherein the first location is apredetermined distance away from the handheld controller along the firstpointing vector.
 5. The system of claim 4, wherein the hardwareprocessor is programmed to: display on the AR display an animation thatappears to come out of the handheld controller towards the firstlocation within the 3D environment of the user.
 6. The system of claim1, wherein the hardware processor is programmed to: display on the ARdisplay the first interactive content object based on a second locationwithin the 3D environment of the user based on movement of the user, andwherein the second location is different from the first location.
 7. Thesystem of claim 1, wherein the hardware processor is programmed to:receive a second indication via the one or more sensors; and stopdisplaying the first interactive content object in response to receivingthe second indication.
 8. The system of claim 1, wherein the hardwareprocessor is programmed to: display on the AR display the firstinteractive content object at a first orientation, and then display onthe AR display the first interactive content object at a secondorientation based on movement of the user, wherein the secondorientation is different from the first orientation.
 9. The system ofclaim 1, wherein the hardware processor is programmed to: receive asecond indication via the one or more sensors; and display on the ARdisplay a second interactive content object based on the secondindication, wherein the second indication is different from the firstindication, and the second interactive content object is different fromthe first interactive content object.
 10. The system of claim 1, thehardware processor is programmed to: receive a second indication via theone or more sensors; and determine the first pointing vector thatindicates the first direction within the 3D environment based on thefirst indication and the second indication, wherein the first indicationindicates an orientation of a handheld controller, and the secondindication indicates an eye gaze direction of the user.
 11. A method ofdisplaying interactive content, the method comprising: receiving a firstindication via one or more sensors of an augmented reality (AR) system;determining a first pointing vector that indicates a first directionwithin a three-dimensional (3D) environment of a user of the AR systembased on the first indication; determining a first location within the3D environment of the user based on the first pointing vector; anddisplaying on a AR display a first interactive content object based onthe first location in the 3D environment of the user, wherein the firstinteractive content object is a control menu or a prism that includesone or more interactive virtual objects.
 12. The method of claim 11,wherein the first indication is a gesture or pose of the user of the ARsystem.
 13. The method of claim 11, wherein the one or more sensorsinclude an outward facing camera, and the first indication is receivedvia the outward facing camera, and wherein the first location is in acenter of a field of view of the user of the AR system.
 14. The methodof claim 11, wherein the first indication is a press of a button of ahandheld controller, wherein the first direction within the 3Denvironment is an indicated direction of the handheld controller, andwherein the first location is a predetermined distance away from thehandheld controller along the first pointing vector.
 15. The method ofclaim 14, further comprising: displaying on the AR display an animationthat appears to come out of the handheld controller towards the firstlocation within the 3D environment of the user.
 16. The method of claim11, further comprising: displaying on the AR display the firstinteractive content object based on a second location within the 3Denvironment of the user based on movement of the user, wherein thesecond location is different from the first location.
 17. The method ofclaim 11, further comprising: receiving a second indication via the oneor more sensors; and stopping display of the first interactive contentobject in response to the receiving the second indication.
 18. Themethod of claim 11, further comprising: displaying on the AR display thefirst interactive content object at a first orientation, and thendisplaying on the AR display the first interactive content object at asecond orientation based on movement of the user, wherein the secondorientation is different from the first orientation.
 19. The method ofclaim 11, further comprising: receiving a second indication via the oneor more sensors; and displaying on the AR display a second interactivecontent object based on the second indication, wherein the secondindication is different from the first indication, and the secondinteractive content object is different from the first interactivecontent object.
 20. The method of claim 11, further comprising:receiving a second indication via the one or more sensors; anddetermining the first pointing vector that indicates the first directionwithin the 3D environment based on the first indication and the secondindication, wherein the first indication indicates an orientation of ahandheld controller, and the second indication indicates an eye gazedirection of the user of the AR system.